ConcurrentRebalancersTest.TwoConcurrentRebalancers/0: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 4
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240503 08:20:40.790913  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14408
W20240503 08:20:40.792987 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:20:40.799022 14528 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:40.883280 14749 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:20:40.884207   773 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:20:40.891991 14555 consensus_peers.cc:485] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.112571 14556 consensus_peers.cc:462] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111)
I20240503 08:20:41.144095 14338 maintenance_manager.cc:419] P 309eed6422ad4e71bcf73f518dc42423: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9242 bytes on disk
I20240503 08:20:41.149700 14337 maintenance_manager.cc:643] P 309eed6422ad4e71bcf73f518dc42423: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s	user 0.005s	sys 0.000s Metrics: {"dirs.queue_time_us":410,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":119,"thread_start_us":445,"threads_started":1}
I20240503 08:20:41.216056  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14408
W20240503 08:20:41.274642 14687 connection.cc:535] client connection to 127.2.49.129:37829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:41.275350  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14539
W20240503 08:20:41.275478 14557 connection.cc:535] client connection to 127.2.49.129:37829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.275832 14557 connection.cc:535] server connection from 127.2.49.129:60669 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.276118 14687 consensus_peers.cc:485] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276392 14817 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276533 14817 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276623 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276939 14557 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:41.282294 14661 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:41.642069  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14539
W20240503 08:20:41.645644 14815 consensus_peers.cc:485] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:41.672734 14685 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:41.700901 14336 connection.cc:535] server connection from 127.2.49.130:44725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.700883 14814 connection.cc:535] client connection to 127.2.49.130:35021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.700910 14816 connection.cc:535] server connection from 127.2.49.130:33947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:41.701182  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14669
W20240503 08:20:41.702270 14814 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:35021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:41.707424 14790 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:20:41.735582 14814 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:35021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.787412 14336 catalog_manager.cc:4573] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 55067e08fb694e90b6366d41cc962095: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:41.815070 14815 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:41.945169 14814 consensus_peers.cc:485] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:35021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:42.074105  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14669
W20240503 08:20:42.133157 14817 connection.cc:535] client connection to 127.2.49.131:45663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:42.133484  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14799
W20240503 08:20:42.134572 14817 consensus_peers.cc:485] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.155920 14817 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.156056 14817 consensus_peers.cc:485] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.156121 14817 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.156193 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:42.156744 14917 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:42.511545   797 raft_consensus.cc:482] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 129a2ac87d7f416db6d47d79bb1c2b5f)
I20240503 08:20:42.511729   797 raft_consensus.cc:504] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:42.512372   797 leader_election.cc:288] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:20:42.513053   687 raft_consensus.cc:482] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 129a2ac87d7f416db6d47d79bb1c2b5f)
I20240503 08:20:42.513206   687 raft_consensus.cc:504] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:42.513726  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14799
I20240503 08:20:42.513799   687 leader_election.cc:288] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:20:42.564114  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14318
I20240503 08:20:42.570080 14376 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:20:42.917398 14335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240503 08:20:42.917982  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14318
2024-05-03T08:20:42Z chronyd exiting
I20240503 08:20:42.992394  2246 test_util.cc:175] -----------------------------------------------
I20240503 08:20:42.992499  2246 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0

Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN      ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:11:07.013054  2246 test_util.cc:268] Using random seed: 1123757965
I20240503 08:11:07.025190  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:07.025288  2246 ts_itest-base.cc:113] --------------
I20240503 08:11:07.025331  2246 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:11:07.025406  2246 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:07.025470  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:07Z Disabled control of system clock
I20240503 08:11:07.051544  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:42891
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:42891
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:07.400899  2258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.401239  2258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.401355  2258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.408854  2258 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.408998  2258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:07.409056  2258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.409147  2258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:07.409214  2258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:07.418972  2258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:42891
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.420318  2258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.422142  2258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.434010  2263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.434598  2264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.435518  2266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.436373  2258 server_base.cc:1034] running on GCE node
I20240503 08:11:07.437234  2258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.438836  2258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.440088  2258 hybrid_clock.cc:648] HybridClock initialized: now 1714723867440040 us; error 40 us; skew 500 ppm
I20240503 08:11:07.440481  2258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.443233  2258 webserver.cc:460] Webserver started at http://127.2.49.190:42357/ using document root <none> and password file <none>
I20240503 08:11:07.444015  2258 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.444142  2258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.444466  2258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.447569  2258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ff194767c5ec4666a800d5120fd0f9bd"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dz87"
I20240503 08:11:07.448455  2258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ff194767c5ec4666a800d5120fd0f9bd"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dz87"
I20240503 08:11:07.454854  2258 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240503 08:11:07.458863  2272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:07.460376  2258 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:11:07.460551  2258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ff194767c5ec4666a800d5120fd0f9bd"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dz87"
I20240503 08:11:07.460759  2258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.475579  2258 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:07.476449  2258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.476699  2258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.496982  2258 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:42891
I20240503 08:11:07.497006  2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:42891 every 8 connection(s)
I20240503 08:11:07.498498  2258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:07.499846  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2258
I20240503 08:11:07.500095  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:07.504181  2324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:07.514613  2324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd: Bootstrap starting.
I20240503 08:11:07.518750  2324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:07.520251  2324 log.cc:824] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:07.523406  2324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd: No bootstrap required, opened a new log
I20240503 08:11:07.531868  2324 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff194767c5ec4666a800d5120fd0f9bd" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42891 } }
I20240503 08:11:07.532342  2324 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:07.532467  2324 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff194767c5ec4666a800d5120fd0f9bd, State: Initialized, Role: FOLLOWER
I20240503 08:11:07.533257  2324 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff194767c5ec4666a800d5120fd0f9bd" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42891 } }
I20240503 08:11:07.533535  2324 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:07.533669  2324 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:07.533803  2324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:07.535948  2324 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff194767c5ec4666a800d5120fd0f9bd" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42891 } }
I20240503 08:11:07.536444  2324 leader_election.cc:302] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff194767c5ec4666a800d5120fd0f9bd; no voters: 
I20240503 08:11:07.537061  2324 leader_election.cc:288] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:11:07.537233  2329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:07.537922  2329 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [term 1 LEADER]: Becoming Leader. State: Replica: ff194767c5ec4666a800d5120fd0f9bd, State: Running, Role: LEADER
I20240503 08:11:07.538990  2329 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff194767c5ec4666a800d5120fd0f9bd" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42891 } }
I20240503 08:11:07.539582  2324 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:07.542372  2330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff194767c5ec4666a800d5120fd0f9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff194767c5ec4666a800d5120fd0f9bd" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42891 } } }
I20240503 08:11:07.542686  2331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff194767c5ec4666a800d5120fd0f9bd. Latest consensus state: current_term: 1 leader_uuid: "ff194767c5ec4666a800d5120fd0f9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff194767c5ec4666a800d5120fd0f9bd" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42891 } } }
I20240503 08:11:07.542899  2330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.542943  2331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.544036  2335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:07.549355  2335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:07.561308  2335 catalog_manager.cc:1347] Generated new cluster ID: 7ed7d4f814fb4528919140181d57bac7
I20240503 08:11:07.561435  2335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:07.577524  2335 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:07.578823  2335 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:07.588912  2335 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ff194767c5ec4666a800d5120fd0f9bd: Generated new TSK 0
I20240503 08:11:07.590377  2335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:07.607409  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 08:11:07.971057  2348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.971372  2348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.971477  2348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.979591  2348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.979743  2348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.979866  2348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:11:07.989249  2348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.990654  2348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.992388  2348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.005520  2353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.006392  2356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.006903  2354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.007674  2348 server_base.cc:1034] running on GCE node
I20240503 08:11:08.008320  2348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.009335  2348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.010632  2348 hybrid_clock.cc:648] HybridClock initialized: now 1714723868010524 us; error 41 us; skew 500 ppm
I20240503 08:11:08.011008  2348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.014050  2348 webserver.cc:460] Webserver started at http://127.2.49.129:43747/ using document root <none> and password file <none>
I20240503 08:11:08.014875  2348 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.015007  2348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.015345  2348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.018129  2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cfc209b43de14ccbbba8e8770cb02d47"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:08.019059  2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cfc209b43de14ccbbba8e8770cb02d47"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:08.024842  2348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:11:08.028692  2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:08.030452  2348 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:08.030823  2348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cfc209b43de14ccbbba8e8770cb02d47"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:08.031045  2348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.051733  2348 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:08.052626  2348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.052894  2348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.053849  2348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.055423  2348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.055523  2348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:08.055645  2348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.055718  2348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:08.090756  2348 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:45423
I20240503 08:11:08.090801  2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:45423 every 8 connection(s)
I20240503 08:11:08.092103  2348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:08.096093  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2348
I20240503 08:11:08.096334  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:08.099872  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:08.106642  2475 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:08.106977  2475 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.107864  2475 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:08.110141  2289 ts_manager.cc:194] Registered new tserver with Master: cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:08.111739  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:48125
W20240503 08:11:08.455425  2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.455741  2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.455881  2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.463460  2479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:08.463606  2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.463786  2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:11:08.473726  2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.475144  2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.476935  2479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.490530  2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.491319  2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.491999  2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.492812  2479 server_base.cc:1034] running on GCE node
I20240503 08:11:08.493476  2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.494511  2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.495738  2479 hybrid_clock.cc:648] HybridClock initialized: now 1714723868495688 us; error 50 us; skew 500 ppm
I20240503 08:11:08.496066  2479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.498620  2479 webserver.cc:460] Webserver started at http://127.2.49.130:44461/ using document root <none> and password file <none>
I20240503 08:11:08.499356  2479 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.499472  2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.499779  2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.502941  2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:08.503742  2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:08.509047  2479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:11:08.512634  2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:08.514101  2479 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:11:08.514312  2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:08.514521  2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.545794  2479 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:08.546754  2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.547045  2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.548107  2479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.549670  2479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.549768  2479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:08.549888  2479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.549962  2479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:08.587299  2479 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:40299
I20240503 08:11:08.587364  2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:40299 every 8 connection(s)
I20240503 08:11:08.588719  2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:08.589821  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2479
I20240503 08:11:08.590024  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:08.593958  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:08.603511  2606 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:08.603890  2606 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.604786  2606 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:08.606536  2289 ts_manager.cc:194] Registered new tserver with Master: a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:08.607789  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:39489
W20240503 08:11:08.952397  2609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.952713  2609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.952817  2609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.960664  2609 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:08.960804  2609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.960923  2609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:11:08.970228  2609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.971630  2609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.973425  2609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.987094  2614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.988304  2617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.989180  2615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.990484  2609 server_base.cc:1034] running on GCE node
I20240503 08:11:08.991202  2609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.992216  2609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.993470  2609 hybrid_clock.cc:648] HybridClock initialized: now 1714723868993405 us; error 64 us; skew 500 ppm
I20240503 08:11:08.993896  2609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.997018  2609 webserver.cc:460] Webserver started at http://127.2.49.131:44785/ using document root <none> and password file <none>
I20240503 08:11:08.997844  2609 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.997988  2609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.998368  2609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.001554  2609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f1443bb164fb4a79a008d9fcf3575b81"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:09.002498  2609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f1443bb164fb4a79a008d9fcf3575b81"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:09.008121  2609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:11:09.011874  2623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.013428  2609 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:11:09.013588  2609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f1443bb164fb4a79a008d9fcf3575b81"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:09.013773  2609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:09.039340  2609 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:09.040258  2609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:09.040532  2609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:09.041517  2609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:09.043210  2609 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:09.043304  2609 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.043401  2609 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:09.043455  2609 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.080469  2609 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:38461
I20240503 08:11:09.080549  2735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:38461 every 8 connection(s)
I20240503 08:11:09.082387  2609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:09.083734  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2609
I20240503 08:11:09.083957  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:09.087462  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:09.097016  2736 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:09.097354  2736 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:09.098727  2736 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:09.100528  2289 ts_manager.cc:194] Registered new tserver with Master: f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:09.101810  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:57861
I20240503 08:11:09.114905  2475 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
W20240503 08:11:09.447058  2739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:09.447360  2739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:09.447500  2739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:09.455680  2739 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:09.455811  2739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:09.455988  2739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:11:09.465778  2739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:09.467144  2739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:09.468897  2739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:09.481909  2744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.482479  2745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.483471  2747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.484369  2739 server_base.cc:1034] running on GCE node
I20240503 08:11:09.485077  2739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:09.486099  2739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:09.487370  2739 hybrid_clock.cc:648] HybridClock initialized: now 1714723869487301 us; error 64 us; skew 500 ppm
I20240503 08:11:09.487702  2739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:09.495347  2739 webserver.cc:460] Webserver started at http://127.2.49.132:34643/ using document root <none> and password file <none>
I20240503 08:11:09.496155  2739 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:09.496261  2739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:09.496687  2739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.500387  2739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "032a50802ce84632bfb57b6a1a840606"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:09.501382  2739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "032a50802ce84632bfb57b6a1a840606"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:09.508066  2739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20240503 08:11:09.511963  2753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.514030  2739 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.000s
I20240503 08:11:09.514282  2739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "032a50802ce84632bfb57b6a1a840606"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:09.514494  2739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:09.529693  2739 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:09.530607  2739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:09.530861  2739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:09.531761  2739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:09.533200  2739 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:09.533293  2739 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.533382  2739 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:09.533437  2739 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.571326  2739 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:34755
I20240503 08:11:09.571388  2865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:34755 every 8 connection(s)
I20240503 08:11:09.572660  2739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:09.577461  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2739
I20240503 08:11:09.577610  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:09.580793  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:0
--local_ip_for_outbound_sockets=127.2.49.133
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:09.586011  2866 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:09.586362  2866 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:09.587543  2866 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:09.589429  2289 ts_manager.cc:194] Registered new tserver with Master: 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:09.590655  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:42041
I20240503 08:11:09.610833  2606 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
W20240503 08:11:09.939472  2869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:09.939863  2869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:09.940011  2869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:09.948050  2869 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:09.948194  2869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:09.948380  2869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:11:09.958323  2869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:09.959720  2869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:09.961513  2869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:09.975055  2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.975579  2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.976449  2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.977337  2869 server_base.cc:1034] running on GCE node
I20240503 08:11:09.977995  2869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:09.979120  2869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:09.980352  2869 hybrid_clock.cc:648] HybridClock initialized: now 1714723869980308 us; error 42 us; skew 500 ppm
I20240503 08:11:09.980682  2869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:09.983157  2869 webserver.cc:460] Webserver started at http://127.2.49.133:40491/ using document root <none> and password file <none>
I20240503 08:11:09.984031  2869 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:09.984155  2869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:09.984472  2869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.987433  2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1cc62a1159dd4742b2a609e4756443f2"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:09.988273  2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1cc62a1159dd4742b2a609e4756443f2"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:09.993908  2869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240503 08:11:09.997583  2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:09.999260  2869 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:09.999454  2869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1cc62a1159dd4742b2a609e4756443f2"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:09.999670  2869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:10.036892  2869 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:10.038331  2869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:10.038637  2869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:10.039711  2869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:10.041350  2869 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:10.041457  2869 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:10.041574  2869 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:10.041642  2869 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:10.078405  2869 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:45149
I20240503 08:11:10.078469  2995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:45149 every 8 connection(s)
I20240503 08:11:10.079885  2869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:10.080801  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2869
I20240503 08:11:10.081009  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:10.085062  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.134:0
--local_ip_for_outbound_sockets=127.2.49.134
--webserver_interface=127.2.49.134
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:10.098870  2996 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:10.099208  2996 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:10.100126  2996 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:10.101848  2289 ts_manager.cc:194] Registered new tserver with Master: 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149)
I20240503 08:11:10.103163  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:56921
I20240503 08:11:10.104713  2736 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
W20240503 08:11:10.446991  2999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:10.447304  2999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:10.447438  2999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:10.455639  2999 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:10.455778  2999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:10.455946  2999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.134
I20240503 08:11:10.465266  2999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.49.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.134
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:10.466740  2999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:10.468482  2999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:10.481528  3004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.482204  3007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.483103  3005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.484035  2999 server_base.cc:1034] running on GCE node
I20240503 08:11:10.484807  2999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:10.485797  2999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:10.487032  2999 hybrid_clock.cc:648] HybridClock initialized: now 1714723870486980 us; error 49 us; skew 500 ppm
I20240503 08:11:10.487372  2999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:10.490327  2999 webserver.cc:460] Webserver started at http://127.2.49.134:45409/ using document root <none> and password file <none>
I20240503 08:11:10.491099  2999 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:10.491219  2999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:10.491531  2999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:10.494292  2999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2fb0e4ffbda54b9db8c5eceb590719e2"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:10.495095  2999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2fb0e4ffbda54b9db8c5eceb590719e2"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:10.500783  2999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.002s
I20240503 08:11:10.504691  3013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:10.506280  2999 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:10.506459  2999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2fb0e4ffbda54b9db8c5eceb590719e2"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:10.506661  2999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:10.541460  2999 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:10.542387  2999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:10.542658  2999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:10.543645  2999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:10.545127  2999 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:10.545226  2999 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:10.545348  2999 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:10.545421  2999 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:10.583065  2999 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.134:34697
I20240503 08:11:10.583149  3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.134:34697 every 8 connection(s)
I20240503 08:11:10.584470  2999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:10.588001  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2999
I20240503 08:11:10.588229  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:10.591645  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.135:0
--local_ip_for_outbound_sockets=127.2.49.135
--webserver_interface=127.2.49.135
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:10.593854  2866 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
I20240503 08:11:10.599378  3126 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:10.599686  3126 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:10.600549  3126 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:10.602073  2289 ts_manager.cc:194] Registered new tserver with Master: 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134:34697)
I20240503 08:11:10.602869  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.134:46335
W20240503 08:11:10.942459  3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:10.942770  3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:10.942914  3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:10.950408  3129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:10.950544  3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:10.950707  3129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.135
I20240503 08:11:10.960238  3129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.49.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.135
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:10.961613  3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:10.963366  3129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:10.976389  3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.976976  3135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.977651  3134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.978689  3129 server_base.cc:1034] running on GCE node
I20240503 08:11:10.979318  3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:10.980257  3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:10.981477  3129 hybrid_clock.cc:648] HybridClock initialized: now 1714723870981437 us; error 33 us; skew 500 ppm
I20240503 08:11:10.981837  3129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:10.984090  3129 webserver.cc:460] Webserver started at http://127.2.49.135:40157/ using document root <none> and password file <none>
I20240503 08:11:10.984849  3129 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:10.984999  3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:10.985299  3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:10.987856  3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e4a4f8044a424243aa9ab82e0896099d"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:10.988612  3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e4a4f8044a424243aa9ab82e0896099d"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:10.995730  3129 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.000s
I20240503 08:11:10.999253  3143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:11.000639  3129 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:11:11.000813  3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e4a4f8044a424243aa9ab82e0896099d"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:11.001011  3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:11.037395  3129 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:11.038309  3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:11.038563  3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:11.039564  3129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:11.041152  3129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:11.041249  3129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:11.041359  3129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:11.041428  3129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:11.077378  3129 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.135:34325
I20240503 08:11:11.077482  3255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.135:34325 every 8 connection(s)
I20240503 08:11:11.078863  3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:11.081172  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 3129
I20240503 08:11:11.081382  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:11:11.086386  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.136:0
--local_ip_for_outbound_sockets=127.2.49.136
--webserver_interface=127.2.49.136
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42891
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:11.098423  3256 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:11.098829  3256 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:11.099716  3256 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:11.101274  2289 ts_manager.cc:194] Registered new tserver with Master: e4a4f8044a424243aa9ab82e0896099d (127.2.49.135:34325)
I20240503 08:11:11.102105  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.135:48333
I20240503 08:11:11.106391  2996 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
W20240503 08:11:11.441517  3259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:11.441830  3259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:11.441932  3259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:11.450095  3259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:11.450250  3259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:11.450383  3259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.136
I20240503 08:11:11.460153  3259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.49.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.136
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:11.461549  3259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:11.463387  3259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:11.476956  3264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.477790  3267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.478403  3265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:11.479153  3259 server_base.cc:1034] running on GCE node
I20240503 08:11:11.479911  3259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:11.480996  3259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:11.482266  3259 hybrid_clock.cc:648] HybridClock initialized: now 1714723871482221 us; error 37 us; skew 500 ppm
I20240503 08:11:11.482600  3259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:11.485004  3259 webserver.cc:460] Webserver started at http://127.2.49.136:42877/ using document root <none> and password file <none>
I20240503 08:11:11.485781  3259 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:11.485881  3259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:11.486307  3259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:11.489006  3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e8ec7129d89f45bc954e578d526e3eaf"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-dz87"
I20240503 08:11:11.489797  3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e8ec7129d89f45bc954e578d526e3eaf"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-dz87"
I20240503 08:11:11.494944  3259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240503 08:11:11.498589  3273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:11.500010  3259 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:11:11.500162  3259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e8ec7129d89f45bc954e578d526e3eaf"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-dz87"
I20240503 08:11:11.500360  3259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:11.525022  3259 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:11.525880  3259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:11.526137  3259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:11.527140  3259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:11.528667  3259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:11.528749  3259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:11.528867  3259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:11.528941  3259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:11.565413  3259 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.136:34127
I20240503 08:11:11.565488  3385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.136:34127 every 8 connection(s)
I20240503 08:11:11.566804  3259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:11:11.568316  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 3259
I20240503 08:11:11.568477  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:11:11.580574  3386 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:11.580884  3386 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:11.581701  3386 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:11.583112  2288 ts_manager.cc:194] Registered new tserver with Master: e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136:34127)
I20240503 08:11:11.584142  2288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.136:35945
I20240503 08:11:11.586573  2246 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:11:11.605628  3126 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
I20240503 08:11:11.609557  2246 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:11:11.609740  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2869
I20240503 08:11:11.618552  2991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:11.702363  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2869
I20240503 08:11:11.718719  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2999
I20240503 08:11:11.726989  3121 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:11.811576  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2999
I20240503 08:11:11.834489  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 3129
I20240503 08:11:11.842810  3251 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:11.924233  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 3129
I20240503 08:11:11.940407  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 3259
I20240503 08:11:11.949129  3381 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:12.032625  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 3259
I20240503 08:11:15.802647  2288 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58240:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:11:15.841367  2541 tablet_service.cc:1467] Processing CreateTablet for tablet e5b84ea2b116462ba589ff50802c536f (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.841138  2539 tablet_service.cc:1467] Processing CreateTablet for tablet cd324aef02344416b643a2421ca46918 (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.843251  2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b84ea2b116462ba589ff50802c536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.843999  2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd324aef02344416b643a2421ca46918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.844918  2801 tablet_service.cc:1467] Processing CreateTablet for tablet e5b84ea2b116462ba589ff50802c536f (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.846041  2540 tablet_service.cc:1467] Processing CreateTablet for tablet 5da6fe49421748a598837fae0d0712e7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.846580  2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da6fe49421748a598837fae0d0712e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.849262  2670 tablet_service.cc:1467] Processing CreateTablet for tablet 5da6fe49421748a598837fae0d0712e7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.851269  2801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b84ea2b116462ba589ff50802c536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.851287  2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da6fe49421748a598837fae0d0712e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.844913  2800 tablet_service.cc:1467] Processing CreateTablet for tablet cd324aef02344416b643a2421ca46918 (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.849234  2671 tablet_service.cc:1467] Processing CreateTablet for tablet e5b84ea2b116462ba589ff50802c536f (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.852154  2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b84ea2b116462ba589ff50802c536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.853600  2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd324aef02344416b643a2421ca46918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.858685  2410 tablet_service.cc:1467] Processing CreateTablet for tablet 5da6fe49421748a598837fae0d0712e7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.860173  2409 tablet_service.cc:1467] Processing CreateTablet for tablet cd324aef02344416b643a2421ca46918 (DEFAULT_TABLE table=rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:15.860513  2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da6fe49421748a598837fae0d0712e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.861056  2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd324aef02344416b643a2421ca46918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:15.863412  3415 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:15.874074  3416 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:15.879230  3416 tablet_bootstrap.cc:654] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.881172  3416 log.cc:824] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:15.881405  3415 tablet_bootstrap.cc:654] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.882656  3415 log.cc:824] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:15.885802  3417 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:15.888006  3418 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:15.890383  3417 tablet_bootstrap.cc:654] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.891934  3417 log.cc:824] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:15.892930  3415 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:15.893435  3415 ts_tablet_manager.cc:1367] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.030s	user 0.008s	sys 0.005s
I20240503 08:11:15.906600  3415 raft_consensus.cc:348] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.907366  3415 raft_consensus.cc:374] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.907547  3415 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.908901  3415 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.910895  3418 tablet_bootstrap.cc:654] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.912709  3418 log.cc:824] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:15.917708  3415 ts_tablet_manager.cc:1398] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.024s	user 0.023s	sys 0.000s
I20240503 08:11:15.918290  3415 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:15.920032  3415 tablet_bootstrap.cc:654] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.920346  3416 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:15.920673  3416 ts_tablet_manager.cc:1367] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.047s	user 0.024s	sys 0.019s
I20240503 08:11:15.920960  3418 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:15.921284  3418 ts_tablet_manager.cc:1367] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.034s	user 0.004s	sys 0.005s
I20240503 08:11:15.926851  3415 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:15.927099  3415 ts_tablet_manager.cc:1367] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.001s
I20240503 08:11:15.928149  3417 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:15.928212  3415 raft_consensus.cc:348] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.928434  3415 raft_consensus.cc:374] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.928468  3417 ts_tablet_manager.cc:1367] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.043s	user 0.018s	sys 0.014s
I20240503 08:11:15.928530  3415 raft_consensus.cc:729] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.928771  3415 consensus_queue.cc:260] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.929392  3415 ts_tablet_manager.cc:1398] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:11:15.929672  3415 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:15.929916  3416 raft_consensus.cc:348] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.930409  3416 raft_consensus.cc:374] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.930500  3416 raft_consensus.cc:729] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.931378  3415 tablet_bootstrap.cc:654] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.931245  3416 consensus_queue.cc:260] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.935817  3416 ts_tablet_manager.cc:1398] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.008s
I20240503 08:11:15.936300  3416 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:15.936312  3415 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:15.936564  3415 ts_tablet_manager.cc:1367] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240503 08:11:15.937335  3415 raft_consensus.cc:348] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.937569  3415 raft_consensus.cc:374] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.937639  3415 raft_consensus.cc:729] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.937891  3415 consensus_queue.cc:260] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.938066  3416 tablet_bootstrap.cc:654] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.938612  3415 ts_tablet_manager.cc:1398] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:11:15.943941  3418 raft_consensus.cc:348] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.944517  3418 raft_consensus.cc:374] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.944633  3418 raft_consensus.cc:729] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.944357  3417 raft_consensus.cc:348] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.944804  3417 raft_consensus.cc:374] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.945072  3417 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.945343  3418 consensus_queue.cc:260] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.945643  3416 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:15.945817  3416 ts_tablet_manager.cc:1367] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.004s
I20240503 08:11:15.945797  3417 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.947163  3416 raft_consensus.cc:348] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.947460  3416 raft_consensus.cc:374] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.947563  3416 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.947633  3418 ts_tablet_manager.cc:1398] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.004s
I20240503 08:11:15.947840  3416 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.948036  3418 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:15.948539  3416 ts_tablet_manager.cc:1398] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.000s
I20240503 08:11:15.949817  3418 tablet_bootstrap.cc:654] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.950153  3417 ts_tablet_manager.cc:1398] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.022s	user 0.016s	sys 0.000s
I20240503 08:11:15.951385  3417 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:15.953266  3417 tablet_bootstrap.cc:654] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:15.953667  3418 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:15.953823  3418 ts_tablet_manager.cc:1367] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 08:11:15.955153  3418 raft_consensus.cc:348] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.955471  3418 raft_consensus.cc:374] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.955577  3418 raft_consensus.cc:729] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.955881  3418 consensus_queue.cc:260] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.956665  3418 ts_tablet_manager.cc:1398] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:11:15.957995  3417 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:15.958180  3417 ts_tablet_manager.cc:1367] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 08:11:15.959116  3417 raft_consensus.cc:348] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.959350  3417 raft_consensus.cc:374] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:15.959453  3417 raft_consensus.cc:729] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:15.959715  3417 consensus_queue.cc:260] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.960333  3417 ts_tablet_manager.cc:1398] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:11:15.977473  3428 raft_consensus.cc:482] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:15.977743  3428 raft_consensus.cc:504] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.979645  3428 leader_election.cc:288] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:15.986776  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:15.986773  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:15.987285  2821 raft_consensus.cc:2454] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:15.987277  2561 raft_consensus.cc:2454] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:15.987938  2365 leader_election.cc:302] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:15.988348  3428 raft_consensus.cc:2781] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:15.988534  3428 raft_consensus.cc:482] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:15.988648  3428 raft_consensus.cc:3037] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:15.990871  3428 raft_consensus.cc:504] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:15.991587  3428 leader_election.cc:288] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Requested vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:15.992113  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:15.992210  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:15.992377  2821 raft_consensus.cc:3037] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:15.992444  2561 raft_consensus.cc:3037] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:15.994699  2561 raft_consensus.cc:2454] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:15.994735  2821 raft_consensus.cc:2454] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:15.995180  2363 leader_election.cc:302] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:15.995505  3428 raft_consensus.cc:2781] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:15.996032  3428 raft_consensus.cc:686] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Becoming Leader. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Running, Role: LEADER
I20240503 08:11:15.996520  3428 consensus_queue.cc:237] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.001426  2288 catalog_manager.cc:5484] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:16.079617  2867 tablet.cc:2369] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:16.089377  2737 tablet.cc:2369] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:16.098520  2607 tablet.cc:2369] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:16.100450  2476 tablet.cc:2369] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:16.152936  3422 raft_consensus.cc:482] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:16.153148  3422 raft_consensus.cc:504] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.154891  3422 leader_election.cc:288] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:16.163010  2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:16.163326  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:16.163527  2691 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 0.
I20240503 08:11:16.163580  2821 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 0.
I20240503 08:11:16.164068  2496 leader_election.cc:302] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
I20240503 08:11:16.164448  3422 raft_consensus.cc:2781] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:16.164578  3422 raft_consensus.cc:482] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:16.164692  3422 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.166817  3422 raft_consensus.cc:504] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.167431  3422 leader_election.cc:288] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 election: Requested vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:16.167923  2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:16.168017  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:16.168090  2691 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.168280  2821 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.170255  2691 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
I20240503 08:11:16.170270  2821 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
I20240503 08:11:16.170707  2496 leader_election.cc:302] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:16.171100  3422 raft_consensus.cc:2781] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:16.171555  3422 raft_consensus.cc:686] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Becoming Leader. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Running, Role: LEADER
I20240503 08:11:16.172130  3422 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.176888  2288 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: term changed from 0 to 1, leader changed from <none> to a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130). New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:16.207116  3429 raft_consensus.cc:482] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:16.207374  3429 raft_consensus.cc:504] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.208984  3429 leader_election.cc:288] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:16.216240  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6fe49421748a598837fae0d0712e7" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:16.216280  2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6fe49421748a598837fae0d0712e7" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47" is_pre_election: true
I20240503 08:11:16.216504  2561 raft_consensus.cc:2454] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:16.216739  2430 raft_consensus.cc:2454] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:16.217029  2624 leader_election.cc:302] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:16.217449  3429 raft_consensus.cc:2781] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:16.217582  3429 raft_consensus.cc:482] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:16.217680  3429 raft_consensus.cc:3037] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.219980  3429 raft_consensus.cc:504] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.220672  3429 leader_election.cc:288] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:16.221179  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6fe49421748a598837fae0d0712e7" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:16.221203  2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6fe49421748a598837fae0d0712e7" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47"
I20240503 08:11:16.221398  2561 raft_consensus.cc:3037] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.221408  2430 raft_consensus.cc:3037] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.223345  2561 raft_consensus.cc:2454] T 5da6fe49421748a598837fae0d0712e7 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:16.223659  2430 raft_consensus.cc:2454] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:16.223750  2624 leader_election.cc:302] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:16.224114  3429 raft_consensus.cc:2781] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:16.224567  3429 raft_consensus.cc:686] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:16.225042  3429 consensus_queue.cc:237] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:16.230026  2288 catalog_manager.cc:5484] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:16.490759  3428 consensus_queue.cc:1035] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:16.501657  3434 consensus_queue.cc:1035] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:16.603703  3444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:16.604058  3444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:16.612217  3444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:11:16.652244  3429 consensus_queue.cc:1035] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:16.658969  3429 consensus_queue.cc:1035] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:16.664124  3422 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:16.670828  3422 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 241.373 ms
  time per row: 0.120686 ms
I20240503 08:11:17.021672  2289 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58240:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:11:17.031526  2410 tablet_service.cc:1467] Processing CreateTablet for tablet a587ae362dc34cde86d1053430ef8e44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.031564  2800 tablet_service.cc:1467] Processing CreateTablet for tablet a587ae362dc34cde86d1053430ef8e44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.032035  2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a587ae362dc34cde86d1053430ef8e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.032042  2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a587ae362dc34cde86d1053430ef8e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.032657  2541 tablet_service.cc:1467] Processing CreateTablet for tablet 390e508ba635499da7e2d5b5ee56e718 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.033067  2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e508ba635499da7e2d5b5ee56e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.035095  2670 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9906f233cd4c848eb58f2330ca70b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.035226  2409 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9906f233cd4c848eb58f2330ca70b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.035616  2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9906f233cd4c848eb58f2330ca70b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.035604  2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9906f233cd4c848eb58f2330ca70b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.036912  2671 tablet_service.cc:1467] Processing CreateTablet for tablet a587ae362dc34cde86d1053430ef8e44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.037333  2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a587ae362dc34cde86d1053430ef8e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.040136  2801 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9906f233cd4c848eb58f2330ca70b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.040726  2801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9906f233cd4c848eb58f2330ca70b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.043143  3508 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:17.043709  2409 tablet_service.cc:1467] Processing CreateTablet for tablet 390e508ba635499da7e2d5b5ee56e718 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.043978  3509 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:17.044294  2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e508ba635499da7e2d5b5ee56e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.045269  3508 tablet_bootstrap.cc:654] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.046294  2669 tablet_service.cc:1467] Processing CreateTablet for tablet 390e508ba635499da7e2d5b5ee56e718 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.046983  2669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e508ba635499da7e2d5b5ee56e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.047122  3509 tablet_bootstrap.cc:654] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.060256  3509 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:17.060556  3509 ts_tablet_manager.cc:1367] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.008s
I20240503 08:11:17.062357  3509 raft_consensus.cc:348] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.063035  3513 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:17.064359  3509 raft_consensus.cc:374] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.064585  3509 raft_consensus.cc:729] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.065169  3509 consensus_queue.cc:260] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.066037  3513 tablet_bootstrap.cc:654] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.066694  3509 ts_tablet_manager.cc:1398] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240503 08:11:17.067034  3509 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:17.068850  3508 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:17.069082  3508 ts_tablet_manager.cc:1367] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.026s	user 0.021s	sys 0.004s
I20240503 08:11:17.069202  3512 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:17.070073  3508 raft_consensus.cc:348] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.070324  3508 raft_consensus.cc:374] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.070425  3508 raft_consensus.cc:729] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.070736  3508 consensus_queue.cc:260] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.071424  3508 ts_tablet_manager.cc:1398] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:17.071786  3508 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:17.073352  3509 tablet_bootstrap.cc:654] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.073529  3508 tablet_bootstrap.cc:654] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.075548  3512 tablet_bootstrap.cc:654] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.076411  3513 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:17.076738  3513 ts_tablet_manager.cc:1367] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.005s
I20240503 08:11:17.077831  3513 raft_consensus.cc:348] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.078214  3513 raft_consensus.cc:374] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.078378  3513 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.078672  3508 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:17.078882  3508 ts_tablet_manager.cc:1367] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240503 08:11:17.078985  3509 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:17.079192  3509 ts_tablet_manager.cc:1367] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.001s
I20240503 08:11:17.079694  3513 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.079820  3508 raft_consensus.cc:348] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.080070  3508 raft_consensus.cc:374] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.080168  3508 raft_consensus.cc:729] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.080473  3508 consensus_queue.cc:260] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.081140  3513 ts_tablet_manager.cc:1398] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:11:17.081183  3508 ts_tablet_manager.cc:1398] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:17.082207  3509 raft_consensus.cc:348] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.082441  3509 raft_consensus.cc:374] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.082510  3509 raft_consensus.cc:729] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.083112  3509 consensus_queue.cc:260] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.083685  3509 ts_tablet_manager.cc:1398] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:11:17.085305  3509 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:17.087002  3509 tablet_bootstrap.cc:654] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.089078  3512 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:17.089254  3512 ts_tablet_manager.cc:1367] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.004s
I20240503 08:11:17.090090  3512 raft_consensus.cc:348] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.090342  3512 raft_consensus.cc:374] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.090484  3512 raft_consensus.cc:729] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.090746  3512 consensus_queue.cc:260] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.091331  3509 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:17.091394  3512 ts_tablet_manager.cc:1398] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:17.091485  3509 ts_tablet_manager.cc:1367] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.003s
I20240503 08:11:17.091668  3512 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:17.093398  3512 tablet_bootstrap.cc:654] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.094345  3509 raft_consensus.cc:348] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.094566  3509 raft_consensus.cc:374] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.094677  3509 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.094931  3509 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.096078  3509 ts_tablet_manager.cc:1398] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:11:17.100714  3512 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:17.101003  3512 ts_tablet_manager.cc:1367] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 08:11:17.101974  3512 raft_consensus.cc:348] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.102298  3512 raft_consensus.cc:374] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.102432  3512 raft_consensus.cc:729] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.102777  3512 consensus_queue.cc:260] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.103976  3512 ts_tablet_manager.cc:1398] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:11:17.104272  3512 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:17.106498  3512 tablet_bootstrap.cc:654] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.109351  3512 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:17.109531  3512 ts_tablet_manager.cc:1367] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.002s
I20240503 08:11:17.110357  3512 raft_consensus.cc:348] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.110560  3512 raft_consensus.cc:374] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.110648  3512 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.110884  3512 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.111953  3512 ts_tablet_manager.cc:1398] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:11:17.154407  3434 raft_consensus.cc:482] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.154639  3434 raft_consensus.cc:504] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.155495  3434 leader_election.cc:288] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:17.157006  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:17.157320  2821 raft_consensus.cc:2454] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:17.157838  2365 leader_election.cc:302] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:17.158216  3434 raft_consensus.cc:2781] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.158304  3434 raft_consensus.cc:482] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.158404  3434 raft_consensus.cc:3037] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.160997  3434 raft_consensus.cc:504] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.161804  3434 leader_election.cc:288] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Requested vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:17.162384  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:17.162577  2821 raft_consensus.cc:3037] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.163357  2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:17.163451  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:17.163654  2691 raft_consensus.cc:2454] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:17.164916  2821 raft_consensus.cc:2454] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:17.165395  2365 leader_election.cc:302] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: f1443bb164fb4a79a008d9fcf3575b81
I20240503 08:11:17.165741  3434 raft_consensus.cc:2781] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.165975  3434 raft_consensus.cc:686] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Becoming Leader. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Running, Role: LEADER
I20240503 08:11:17.166289  3434 consensus_queue.cc:237] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:17.169307  2289 catalog_manager.cc:5484] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:17.172577  3442 raft_consensus.cc:482] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.172806  3442 raft_consensus.cc:504] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.173815  3442 leader_election.cc:288] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:17.174410  2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a587ae362dc34cde86d1053430ef8e44" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47" is_pre_election: true
I20240503 08:11:17.174741  2430 raft_consensus.cc:2454] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:17.175240  2626 leader_election.cc:302] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc209b43de14ccbbba8e8770cb02d47, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:17.175541  3442 raft_consensus.cc:2781] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.175699  3442 raft_consensus.cc:482] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.175815  3442 raft_consensus.cc:3037] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.178814  3442 raft_consensus.cc:504] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.179481  3442 leader_election.cc:288] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:17.180078  2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a587ae362dc34cde86d1053430ef8e44" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47"
I20240503 08:11:17.180373  2430 raft_consensus.cc:3037] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.182505  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a587ae362dc34cde86d1053430ef8e44" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:17.182773  2821 raft_consensus.cc:2454] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:17.182765  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a587ae362dc34cde86d1053430ef8e44" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:17.182982  2820 raft_consensus.cc:3037] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.183025  2430 raft_consensus.cc:2454] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:17.183422  2626 leader_election.cc:302] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc209b43de14ccbbba8e8770cb02d47, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:17.183712  3442 raft_consensus.cc:2781] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.183863  3442 raft_consensus.cc:686] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:17.184177  3442 consensus_queue.cc:237] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:17.185204  2820 raft_consensus.cc:2454] T a587ae362dc34cde86d1053430ef8e44 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:17.187055  2289 catalog_manager.cc:5484] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:17.283671  3434 raft_consensus.cc:482] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.283892  3434 raft_consensus.cc:504] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.284826  3434 leader_election.cc:288] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:17.285369  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:17.285388  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:17.285588  2560 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:17.285614  2690 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:17.286089  2363 leader_election.cc:302] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:17.286461  3434 raft_consensus.cc:2781] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.286604  3434 raft_consensus.cc:482] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.286679  3434 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.289335  3434 raft_consensus.cc:504] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.290040  3434 leader_election.cc:288] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:17.290653  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:17.290674  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:17.290865  2560 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.290865  2690 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.293210  2560 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:17.293313  2690 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:17.293792  2363 leader_election.cc:302] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:17.294109  3434 raft_consensus.cc:2781] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.294379  3434 raft_consensus.cc:686] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Becoming Leader. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Running, Role: LEADER
I20240503 08:11:17.294672  3434 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:17.297499  2289 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:17.639063  3434 consensus_queue.cc:1035] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.645817  2690 raft_consensus.cc:3037] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.649451  3434 consensus_queue.cc:1035] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:17.720782  3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:17.721266  3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:17.729552  3527 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:11:17.792430  3442 consensus_queue.cc:1035] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.818818  3559 consensus_queue.cc:1035] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.823748  3531 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.832094  3428 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 235.905 ms
  time per row: 0.117953 ms
I20240503 08:11:18.121268  2285 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58240:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:11:18.131932  2541 tablet_service.cc:1467] Processing CreateTablet for tablet 4d06805616834a3ca83ed6bb5d6ad320 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.132175  2801 tablet_service.cc:1467] Processing CreateTablet for tablet 4d06805616834a3ca83ed6bb5d6ad320 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.132437  2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d06805616834a3ca83ed6bb5d6ad320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.132622  2801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d06805616834a3ca83ed6bb5d6ad320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.132500  2409 tablet_service.cc:1467] Processing CreateTablet for tablet 4d06805616834a3ca83ed6bb5d6ad320 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.132925  2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d06805616834a3ca83ed6bb5d6ad320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.135604  2670 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb470b78894940968683d4bf78bb64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.135916  2410 tablet_service.cc:1467] Processing CreateTablet for tablet a62c007e061d4a96b4a3c8633bd29ee5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.136085  2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb470b78894940968683d4bf78bb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.136317  2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62c007e061d4a96b4a3c8633bd29ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.139369  2800 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb470b78894940968683d4bf78bb64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.139820  2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb470b78894940968683d4bf78bb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.142897  2799 tablet_service.cc:1467] Processing CreateTablet for tablet a62c007e061d4a96b4a3c8633bd29ee5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.143458  2799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62c007e061d4a96b4a3c8633bd29ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.147204  2539 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb470b78894940968683d4bf78bb64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.148000  2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb470b78894940968683d4bf78bb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.151615  2540 tablet_service.cc:1467] Processing CreateTablet for tablet a62c007e061d4a96b4a3c8633bd29ee5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.152451  2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62c007e061d4a96b4a3c8633bd29ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.153704  3593 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:18.156297  3593 tablet_bootstrap.cc:654] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.158089  3592 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:18.161264  3592 tablet_bootstrap.cc:654] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.165403  3591 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:18.167523  3591 tablet_bootstrap.cc:654] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.167759  3592 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:18.167996  3592 ts_tablet_manager.cc:1367] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.003s
I20240503 08:11:18.168448  3596 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:18.170413  3596 tablet_bootstrap.cc:654] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.171423  3592 raft_consensus.cc:348] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.172703  3592 raft_consensus.cc:374] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.176569  3592 raft_consensus.cc:729] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.178098  3593 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:18.178406  3593 ts_tablet_manager.cc:1367] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.001s
I20240503 08:11:18.178944  3591 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:18.179118  3591 ts_tablet_manager.cc:1367] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.008s
I20240503 08:11:18.179378  3596 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:18.179548  3596 ts_tablet_manager.cc:1367] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.004s
I20240503 08:11:18.179525  3593 raft_consensus.cc:348] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.179746  3593 raft_consensus.cc:374] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.179944  3593 raft_consensus.cc:729] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.180114  3591 raft_consensus.cc:348] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.180358  3591 raft_consensus.cc:374] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.180456  3591 raft_consensus.cc:729] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.180303  3593 consensus_queue.cc:260] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.180446  3596 raft_consensus.cc:348] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.180681  3596 raft_consensus.cc:374] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.180799  3596 raft_consensus.cc:729] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.180697  3591 consensus_queue.cc:260] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.181090  3596 consensus_queue.cc:260] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.181418  3591 ts_tablet_manager.cc:1398] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:11:18.181722  3591 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:18.181810  3596 ts_tablet_manager.cc:1398] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:11:18.182102  3596 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:18.183533  3591 tablet_bootstrap.cc:654] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.183506  3593 ts_tablet_manager.cc:1398] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240503 08:11:18.184044  3596 tablet_bootstrap.cc:654] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.183993  3592 consensus_queue.cc:260] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.184731  3592 ts_tablet_manager.cc:1398] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.017s	user 0.001s	sys 0.001s
I20240503 08:11:18.185027  3592 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:18.187078  3592 tablet_bootstrap.cc:654] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.188769  3596 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:18.188983  3596 ts_tablet_manager.cc:1367] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240503 08:11:18.189872  3596 raft_consensus.cc:348] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.190127  3596 raft_consensus.cc:374] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.190320  3596 raft_consensus.cc:729] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.191820  3596 consensus_queue.cc:260] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.192626  3596 ts_tablet_manager.cc:1398] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:11:18.193022  3596 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:18.200196  3591 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:18.200346  3596 tablet_bootstrap.cc:654] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.200546  3591 ts_tablet_manager.cc:1367] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.008s
I20240503 08:11:18.201452  3591 raft_consensus.cc:348] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.201812  3591 raft_consensus.cc:374] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.201947  3591 raft_consensus.cc:729] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.202356  3591 consensus_queue.cc:260] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.203732  3591 ts_tablet_manager.cc:1398] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:11:18.205809  3592 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:18.206065  3592 ts_tablet_manager.cc:1367] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.010s
I20240503 08:11:18.206851  3592 raft_consensus.cc:348] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.207055  3592 raft_consensus.cc:374] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.207139  3592 raft_consensus.cc:729] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.207440  3592 consensus_queue.cc:260] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.207800  3596 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:18.207980  3592 ts_tablet_manager.cc:1398] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:11:18.207979  3596 ts_tablet_manager.cc:1367] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.003s
I20240503 08:11:18.208351  3592 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:18.208876  3596 raft_consensus.cc:348] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.209091  3596 raft_consensus.cc:374] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.209234  3596 raft_consensus.cc:729] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.209530  3596 consensus_queue.cc:260] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.209766  3592 tablet_bootstrap.cc:654] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.210285  3596 ts_tablet_manager.cc:1398] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 08:11:18.211899  3592 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:18.212044  3592 ts_tablet_manager.cc:1367] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:11:18.212981  3592 raft_consensus.cc:348] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.213272  3592 raft_consensus.cc:374] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.213392  3592 raft_consensus.cc:729] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.213616  3592 consensus_queue.cc:260] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.214131  3592 ts_tablet_manager.cc:1398] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:18.250802  3428 raft_consensus.cc:482] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.251014  3428 raft_consensus.cc:504] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.251695  3428 leader_election.cc:288] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:18.252259  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:18.252357  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:18.252549  2820 raft_consensus.cc:2454] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:18.252550  2561 raft_consensus.cc:2454] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:18.252969  2363 leader_election.cc:302] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:18.253358  3428 raft_consensus.cc:2781] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.253449  3428 raft_consensus.cc:482] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.253571  3428 raft_consensus.cc:3037] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.255565  3428 raft_consensus.cc:504] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.256134  3428 leader_election.cc:288] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:18.256597  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:18.256620  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:18.256815  2561 raft_consensus.cc:3037] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.256826  2820 raft_consensus.cc:3037] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.258728  2820 raft_consensus.cc:2454] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:18.258901  2561 raft_consensus.cc:2454] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:18.259251  2365 leader_election.cc:302] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:18.259611  3428 raft_consensus.cc:2781] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:18.259783  3428 raft_consensus.cc:686] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Becoming Leader. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Running, Role: LEADER
I20240503 08:11:18.260092  3428 consensus_queue.cc:237] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.263095  2285 catalog_manager.cc:5484] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:18.353847  3528 raft_consensus.cc:482] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.354081  3528 raft_consensus.cc:504] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.355904  3528 leader_election.cc:288] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:18.362147  3528 raft_consensus.cc:482] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.362413  3528 raft_consensus.cc:504] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.363263  3528 leader_election.cc:288] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:18.363291  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d06805616834a3ca83ed6bb5d6ad320" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:18.363322  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d06805616834a3ca83ed6bb5d6ad320" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47" is_pre_election: true
I20240503 08:11:18.363579  2429 raft_consensus.cc:2454] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 0.
I20240503 08:11:18.363631  2561 raft_consensus.cc:2454] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 0.
I20240503 08:11:18.364064  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:18.364203  2756 leader_election.cc:302] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:18.364348  2561 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 0.
I20240503 08:11:18.364750  3528 raft_consensus.cc:2781] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.364878  3528 raft_consensus.cc:482] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.365003  3528 raft_consensus.cc:3037] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.365185  2754 leader_election.cc:302] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
I20240503 08:11:18.365996  3608 raft_consensus.cc:2781] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.366122  3608 raft_consensus.cc:482] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.366287  3608 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.368175  3528 raft_consensus.cc:504] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.369407  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d06805616834a3ca83ed6bb5d6ad320" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:18.369675  2561 raft_consensus.cc:3037] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.370000  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d06805616834a3ca83ed6bb5d6ad320" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47"
I20240503 08:11:18.370256  2429 raft_consensus.cc:3037] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.370278  3608 raft_consensus.cc:504] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.371006  3608 leader_election.cc:288] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 election: Requested vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:18.371594  3528 leader_election.cc:288] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:18.372225  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:18.372450  2560 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.372722  2561 raft_consensus.cc:2454] T 4d06805616834a3ca83ed6bb5d6ad320 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:18.372747  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:18.372993  2690 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 0.
I20240503 08:11:18.373154  2429 raft_consensus.cc:2454] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:18.373162  2754 leader_election.cc:302] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
I20240503 08:11:18.373353  2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:18.373476  3528 raft_consensus.cc:2781] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:18.373556  2691 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.373648  3528 raft_consensus.cc:686] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Becoming Leader. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:18.374181  3528 consensus_queue.cc:237] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:18.375172  2560 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:18.375676  2754 leader_election.cc:302] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
I20240503 08:11:18.376052  2691 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:18.376503  3609 raft_consensus.cc:2781] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:18.376708  3609 raft_consensus.cc:686] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Becoming Leader. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:18.377032  3609 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:18.379534  2285 catalog_manager.cc:5484] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 reported cstate change: term changed from 0 to 1, leader changed from <none> to 032a50802ce84632bfb57b6a1a840606 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:18.384491  2285 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 reported cstate change: term changed from 0 to 1, leader changed from <none> to 032a50802ce84632bfb57b6a1a840606 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:18.653435  3428 consensus_queue.cc:1035] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.660836  3428 consensus_queue.cc:1035] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.763764  3609 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.768997  3528 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.779460  3609 consensus_queue.cc:1035] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:18.785100  3608 consensus_queue.cc:1035] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:18.799937  3612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.800287  3612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.807997  3612 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 252.229 ms
  time per row: 0.126115 ms
I20240503 08:11:19.215246  2282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58240:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:11:19.225219  2670 tablet_service.cc:1467] Processing CreateTablet for tablet 45e97b1190e742f1817c55b578728657 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.225709  2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e97b1190e742f1817c55b578728657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.225750  2540 tablet_service.cc:1467] Processing CreateTablet for tablet 45e97b1190e742f1817c55b578728657 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.225891  2409 tablet_service.cc:1467] Processing CreateTablet for tablet 45e97b1190e742f1817c55b578728657 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.226213  2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e97b1190e742f1817c55b578728657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.226356  2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e97b1190e742f1817c55b578728657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.229792  2669 tablet_service.cc:1467] Processing CreateTablet for tablet 22756ba18d2f49999f02ac878b8bf03d (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.229903  2800 tablet_service.cc:1467] Processing CreateTablet for tablet 22756ba18d2f49999f02ac878b8bf03d (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.230291  2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22756ba18d2f49999f02ac878b8bf03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.230300  2669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22756ba18d2f49999f02ac878b8bf03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.230563  2410 tablet_service.cc:1467] Processing CreateTablet for tablet 22756ba18d2f49999f02ac878b8bf03d (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.231945  2539 tablet_service.cc:1467] Processing CreateTablet for tablet 26168740892c49e2a61445c79ecdb54c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.231945  2671 tablet_service.cc:1467] Processing CreateTablet for tablet 26168740892c49e2a61445c79ecdb54c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.232342  2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26168740892c49e2a61445c79ecdb54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.232355  2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26168740892c49e2a61445c79ecdb54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.235213  2799 tablet_service.cc:1467] Processing CreateTablet for tablet 26168740892c49e2a61445c79ecdb54c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:19.235749  2799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26168740892c49e2a61445c79ecdb54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.238513  2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22756ba18d2f49999f02ac878b8bf03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.250783  3673 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:19.252928  3673 tablet_bootstrap.cc:654] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.254971  3675 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:19.258039  3675 tablet_bootstrap.cc:654] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.258069  3674 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:19.260751  3672 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:19.261191  3674 tablet_bootstrap.cc:654] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.262598  3672 tablet_bootstrap.cc:654] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.263958  3673 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:19.264252  3673 ts_tablet_manager.cc:1367] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.004s
I20240503 08:11:19.264945  3675 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:19.265110  3675 ts_tablet_manager.cc:1367] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240503 08:11:19.265460  3673 raft_consensus.cc:348] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.265767  3673 raft_consensus.cc:374] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.265911  3673 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.265866  3675 raft_consensus.cc:348] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.266053  3675 raft_consensus.cc:374] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.266152  3675 raft_consensus.cc:729] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.266336  3673 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.266489  3675 consensus_queue.cc:260] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.271543  3673 ts_tablet_manager.cc:1398] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 08:11:19.271535  3675 ts_tablet_manager.cc:1398] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 08:11:19.275748  3673 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:19.275811  3675 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:19.277992  3675 tablet_bootstrap.cc:654] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.279496  3673 tablet_bootstrap.cc:654] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.280138  3674 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:19.280380  3674 ts_tablet_manager.cc:1367] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.022s	user 0.007s	sys 0.014s
I20240503 08:11:19.281576  3674 raft_consensus.cc:348] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.281980  3674 raft_consensus.cc:374] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.282128  3674 raft_consensus.cc:729] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.282923  3674 consensus_queue.cc:260] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.283967  3674 ts_tablet_manager.cc:1398] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:19.284412  3674 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:19.287539  3674 tablet_bootstrap.cc:654] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.289788  3675 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:19.289835  3672 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:19.289986  3673 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:19.290019  3675 ts_tablet_manager.cc:1367] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.009s
I20240503 08:11:19.290031  3672 ts_tablet_manager.cc:1367] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.012s
I20240503 08:11:19.290227  3673 ts_tablet_manager.cc:1367] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.009s
I20240503 08:11:19.290827  3675 raft_consensus.cc:348] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.290895  3672 raft_consensus.cc:348] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.291029  3675 raft_consensus.cc:374] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.291121  3672 raft_consensus.cc:374] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.291110  3675 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.291049  3673 raft_consensus.cc:348] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.291321  3673 raft_consensus.cc:374] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.291332  3672 raft_consensus.cc:729] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.291416  3673 raft_consensus.cc:729] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.291378  3675 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.291596  3672 consensus_queue.cc:260] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.291657  3673 consensus_queue.cc:260] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.292025  3675 ts_tablet_manager.cc:1398] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:19.293094  3673 ts_tablet_manager.cc:1398] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:19.293568  3672 ts_tablet_manager.cc:1398] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:19.293911  3672 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:19.294814  3674 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:19.295028  3674 ts_tablet_manager.cc:1367] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240503 08:11:19.295444  3672 tablet_bootstrap.cc:654] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.296046  3674 raft_consensus.cc:348] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.296501  3674 raft_consensus.cc:374] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.296617  3674 raft_consensus.cc:729] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.296921  3674 consensus_queue.cc:260] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.299194  3674 ts_tablet_manager.cc:1398] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:19.301831  3672 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:19.302024  3672 ts_tablet_manager.cc:1367] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.002s
I20240503 08:11:19.302676  3672 raft_consensus.cc:348] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.302870  3672 raft_consensus.cc:374] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.302980  3672 raft_consensus.cc:729] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.303251  3672 consensus_queue.cc:260] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.303697  3672 ts_tablet_manager.cc:1398] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:19.303992  3672 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:19.305274  3672 tablet_bootstrap.cc:654] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.306950  3672 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:19.307086  3672 ts_tablet_manager.cc:1367] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:11:19.307678  3672 raft_consensus.cc:348] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.307845  3672 raft_consensus.cc:374] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.307942  3672 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.308180  3672 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.308665  3672 ts_tablet_manager.cc:1398] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:11:19.328986  3422 raft_consensus.cc:482] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:19.329227  3422 raft_consensus.cc:504] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.329869  3422 leader_election.cc:288] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:19.330415  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:19.330463  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:19.330669  2821 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 0.
I20240503 08:11:19.330675  2690 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 0.
I20240503 08:11:19.331054  2496 leader_election.cc:302] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:19.331354  3422 raft_consensus.cc:2781] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:19.331435  3422 raft_consensus.cc:482] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:19.331506  3422 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.333391  3422 raft_consensus.cc:504] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.333923  3422 leader_election.cc:288] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 election: Requested vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:19.334499  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:19.334580  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:19.334722  2821 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.334779  2690 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.336716  2690 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
I20240503 08:11:19.336864  2821 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
I20240503 08:11:19.337193  2496 leader_election.cc:302] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:19.337473  3422 raft_consensus.cc:2781] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.338014  3422 raft_consensus.cc:686] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Becoming Leader. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Running, Role: LEADER
I20240503 08:11:19.338372  3422 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:19.341500  2280 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: term changed from 0 to 1, leader changed from <none> to a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130). New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:19.353977  3434 raft_consensus.cc:482] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:19.354218  3434 raft_consensus.cc:504] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.354818  3434 leader_election.cc:288] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:19.355515  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:19.355506  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:19.355715  2821 raft_consensus.cc:2454] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:19.355763  2690 raft_consensus.cc:2454] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:19.356173  2365 leader_election.cc:302] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:19.356468  3434 raft_consensus.cc:2781] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:19.356570  3434 raft_consensus.cc:482] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:19.356674  3434 raft_consensus.cc:3037] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.358683  3434 raft_consensus.cc:504] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.359360  3434 leader_election.cc:288] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Requested vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:19.359913  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:19.359925  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:19.360090  2821 raft_consensus.cc:3037] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.360105  2690 raft_consensus.cc:3037] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.362015  2821 raft_consensus.cc:2454] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:19.362128  2690 raft_consensus.cc:2454] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:19.362464  2365 leader_election.cc:302] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:19.362794  3434 raft_consensus.cc:2781] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.362972  3434 raft_consensus.cc:686] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Becoming Leader. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Running, Role: LEADER
I20240503 08:11:19.363216  3434 consensus_queue.cc:237] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:19.366286  2280 catalog_manager.cc:5484] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:19.518330  3558 raft_consensus.cc:482] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:19.518525  3558 raft_consensus.cc:504] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.519351  3558 leader_election.cc:288] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:19.519934  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e97b1190e742f1817c55b578728657" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:19.519939  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e97b1190e742f1817c55b578728657" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47" is_pre_election: true
I20240503 08:11:19.520205  2561 raft_consensus.cc:2454] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:19.520180  2429 raft_consensus.cc:2454] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:19.520754  2624 leader_election.cc:302] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:19.525862  3558 raft_consensus.cc:2781] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:19.526041  3558 raft_consensus.cc:482] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:19.526175  3558 raft_consensus.cc:3037] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.528293  3558 raft_consensus.cc:504] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.528963  3558 leader_election.cc:288] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:19.529523  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e97b1190e742f1817c55b578728657" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47"
I20240503 08:11:19.529542  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e97b1190e742f1817c55b578728657" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:19.529723  2561 raft_consensus.cc:3037] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.529754  2429 raft_consensus.cc:3037] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.531780  2429 raft_consensus.cc:2454] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:19.531790  2561 raft_consensus.cc:2454] T 45e97b1190e742f1817c55b578728657 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:19.532227  2624 leader_election.cc:302] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:19.532528  3558 raft_consensus.cc:2781] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.532977  3558 raft_consensus.cc:686] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:19.533298  3558 consensus_queue.cc:237] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:19.536264  2280 catalog_manager.cc:5484] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:19.773177  3434 consensus_queue.cc:1035] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.779361  3434 consensus_queue.cc:1035] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.821902  3422 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.832348  3422 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:19.903827  3690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:19.904279  3690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:19.911845  3690 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:19.935770  3558 consensus_queue.cc:1035] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.941648  3558 consensus_queue.cc:1035] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 302.76 ms
  time per row: 0.15138 ms
I20240503 08:11:20.363575  2289 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58240:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:11:20.373275  2409 tablet_service.cc:1467] Processing CreateTablet for tablet 5268a78550334d9b828888bc358ea695 (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.373366  2540 tablet_service.cc:1467] Processing CreateTablet for tablet 5268a78550334d9b828888bc358ea695 (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.373756  2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5268a78550334d9b828888bc358ea695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.373768  2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5268a78550334d9b828888bc358ea695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.373610  2670 tablet_service.cc:1467] Processing CreateTablet for tablet 5268a78550334d9b828888bc358ea695 (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.374089  2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5268a78550334d9b828888bc358ea695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.375723  2799 tablet_service.cc:1467] Processing CreateTablet for tablet bbc60d8fc83f4cfab23bffcbd1eb72db (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.376170  2799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc60d8fc83f4cfab23bffcbd1eb72db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.378615  2539 tablet_service.cc:1467] Processing CreateTablet for tablet e9aa470143b74329b6f63d3ff1b1ec86 (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.379066  2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9aa470143b74329b6f63d3ff1b1ec86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.380623  2800 tablet_service.cc:1467] Processing CreateTablet for tablet e9aa470143b74329b6f63d3ff1b1ec86 (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.381049  2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9aa470143b74329b6f63d3ff1b1ec86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.383443  2410 tablet_service.cc:1467] Processing CreateTablet for tablet e9aa470143b74329b6f63d3ff1b1ec86 (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.383953  2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9aa470143b74329b6f63d3ff1b1ec86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.387007  2671 tablet_service.cc:1467] Processing CreateTablet for tablet bbc60d8fc83f4cfab23bffcbd1eb72db (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.387554  2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc60d8fc83f4cfab23bffcbd1eb72db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.390823  3764 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:20.390870  2408 tablet_service.cc:1467] Processing CreateTablet for tablet bbc60d8fc83f4cfab23bffcbd1eb72db (DEFAULT_TABLE table=rebalance_test_table_4 [id=edf19b2f9d2241c8a6dd396f72d068a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.391507  2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc60d8fc83f4cfab23bffcbd1eb72db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.393427  3764 tablet_bootstrap.cc:654] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.394140  3763 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:20.396173  3763 tablet_bootstrap.cc:654] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.403517  3762 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:20.406996  3762 tablet_bootstrap.cc:654] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.408707  3767 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:20.409976  3764 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:20.410223  3763 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:20.410252  3764 ts_tablet_manager.cc:1367] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.010s
I20240503 08:11:20.410377  3763 ts_tablet_manager.cc:1367] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.008s
I20240503 08:11:20.411132  3763 raft_consensus.cc:348] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.411094  3764 raft_consensus.cc:348] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.411314  3763 raft_consensus.cc:374] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.411319  3764 raft_consensus.cc:374] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.411401  3764 raft_consensus.cc:729] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.411401  3763 raft_consensus.cc:729] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.411620  3764 consensus_queue.cc:260] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.411620  3763 consensus_queue.cc:260] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.412147  3764 ts_tablet_manager.cc:1398] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:11:20.412196  3763 ts_tablet_manager.cc:1398] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:11:20.412420  3764 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606: Bootstrap starting.
I20240503 08:11:20.412428  3763 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e: Bootstrap starting.
I20240503 08:11:20.414021  3764 tablet_bootstrap.cc:654] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.414022  3763 tablet_bootstrap.cc:654] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.414041  3762 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:20.414211  3762 ts_tablet_manager.cc:1367] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.004s
I20240503 08:11:20.415053  3762 raft_consensus.cc:348] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.415277  3762 raft_consensus.cc:374] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.415369  3762 raft_consensus.cc:729] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.415588  3762 consensus_queue.cc:260] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.417479  3762 ts_tablet_manager.cc:1398] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:20.417773  3762 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81: Bootstrap starting.
I20240503 08:11:20.418100  3767 tablet_bootstrap.cc:654] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.419744  3762 tablet_bootstrap.cc:654] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.425371  3764 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606: No bootstrap required, opened a new log
I20240503 08:11:20.425755  3764 ts_tablet_manager.cc:1367] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.008s
I20240503 08:11:20.426962  3764 raft_consensus.cc:348] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.427424  3764 raft_consensus.cc:374] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.427570  3764 raft_consensus.cc:729] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.428011  3764 consensus_queue.cc:260] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.428711  3762 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81: No bootstrap required, opened a new log
I20240503 08:11:20.428726  3767 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:20.428877  3764 ts_tablet_manager.cc:1398] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:20.428884  3767 ts_tablet_manager.cc:1367] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.003s
I20240503 08:11:20.428953  3762 ts_tablet_manager.cc:1367] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.003s
I20240503 08:11:20.429738  3762 raft_consensus.cc:348] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.429756  3767 raft_consensus.cc:348] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.429952  3767 raft_consensus.cc:374] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.429972  3762 raft_consensus.cc:374] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.430043  3762 raft_consensus.cc:729] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.430080  3767 raft_consensus.cc:729] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.429562  3763 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e: No bootstrap required, opened a new log
I20240503 08:11:20.430361  3767 consensus_queue.cc:260] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.430510  3763 ts_tablet_manager.cc:1367] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.005s
I20240503 08:11:20.430337  3762 consensus_queue.cc:260] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.431365  3762 ts_tablet_manager.cc:1398] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:11:20.431929  3767 ts_tablet_manager.cc:1398] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:20.432462  3763 raft_consensus.cc:348] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.433293  3763 raft_consensus.cc:374] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.433337  3767 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:20.433380  3763 raft_consensus.cc:729] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.433640  3763 consensus_queue.cc:260] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.434275  3763 ts_tablet_manager.cc:1398] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 08:11:20.435917  3767 tablet_bootstrap.cc:654] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.439272  3767 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:20.439412  3767 ts_tablet_manager.cc:1367] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240503 08:11:20.439996  3767 raft_consensus.cc:348] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.440192  3767 raft_consensus.cc:374] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.440277  3767 raft_consensus.cc:729] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.440531  3767 consensus_queue.cc:260] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.440953  3767 ts_tablet_manager.cc:1398] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:11:20.441301  3767 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:20.442996  3767 tablet_bootstrap.cc:654] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.446624  3767 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: No bootstrap required, opened a new log
I20240503 08:11:20.446832  3767 ts_tablet_manager.cc:1367] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 08:11:20.447587  3767 raft_consensus.cc:348] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.447767  3767 raft_consensus.cc:374] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.447851  3767 raft_consensus.cc:729] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.448076  3767 consensus_queue.cc:260] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.448552  3767 ts_tablet_manager.cc:1398] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:11:20.507079  3742 raft_consensus.cc:482] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.507257  3742 raft_consensus.cc:504] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.508023  3742 leader_election.cc:288] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:20.508482  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:20.508627  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:20.508776  2561 raft_consensus.cc:2454] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:20.508852  2690 raft_consensus.cc:2454] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 0.
I20240503 08:11:20.509248  2363 leader_election.cc:302] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:20.509590  3742 raft_consensus.cc:2781] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.509721  3742 raft_consensus.cc:482] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.509809  3742 raft_consensus.cc:3037] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.512869  3742 raft_consensus.cc:504] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.513660  3742 leader_election.cc:288] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:20.514240  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:20.514272  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "cfc209b43de14ccbbba8e8770cb02d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:20.514487  2561 raft_consensus.cc:3037] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.514498  2690 raft_consensus.cc:3037] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.516378  2561 raft_consensus.cc:2454] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:20.516996  2363 leader_election.cc:302] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:20.517333  3742 raft_consensus.cc:2781] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.517508  2690 raft_consensus.cc:2454] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc209b43de14ccbbba8e8770cb02d47 in term 1.
I20240503 08:11:20.517513  3742 raft_consensus.cc:686] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Becoming Leader. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Running, Role: LEADER
I20240503 08:11:20.518235  3742 consensus_queue.cc:237] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:20.521394  2282 catalog_manager.cc:5484] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:20.623399  3422 raft_consensus.cc:482] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.623592  3422 raft_consensus.cc:504] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.624344  3422 leader_election.cc:288] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:20.625802  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9aa470143b74329b6f63d3ff1b1ec86" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:20.626075  2820 raft_consensus.cc:2454] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 0.
I20240503 08:11:20.626605  2496 leader_election.cc:302] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
I20240503 08:11:20.626904  3422 raft_consensus.cc:2781] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.627002  3422 raft_consensus.cc:482] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.627099  3422 raft_consensus.cc:3037] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.630465  3422 raft_consensus.cc:504] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.631186  3422 leader_election.cc:288] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:20.631227  2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9aa470143b74329b6f63d3ff1b1ec86" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47" is_pre_election: true
I20240503 08:11:20.631486  2430 raft_consensus.cc:2454] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 0.
I20240503 08:11:20.631701  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9aa470143b74329b6f63d3ff1b1ec86" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47"
I20240503 08:11:20.631870  2429 raft_consensus.cc:3037] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.631925  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9aa470143b74329b6f63d3ff1b1ec86" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:20.632083  2820 raft_consensus.cc:3037] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.633738  2429 raft_consensus.cc:2454] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
I20240503 08:11:20.633915  2820 raft_consensus.cc:2454] T e9aa470143b74329b6f63d3ff1b1ec86 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
I20240503 08:11:20.634434  2496 leader_election.cc:302] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47; no voters: 
I20240503 08:11:20.634773  3422 raft_consensus.cc:2781] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.634922  3422 raft_consensus.cc:686] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Becoming Leader. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Running, Role: LEADER
I20240503 08:11:20.635243  3422 consensus_queue.cc:237] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:20.638327  2282 catalog_manager.cc:5484] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: term changed from 0 to 1, leader changed from <none> to a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130). New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:20.745210  3688 raft_consensus.cc:482] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.745457  3688 raft_consensus.cc:504] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.746248  3688 leader_election.cc:288] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:20.746791  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc60d8fc83f4cfab23bffcbd1eb72db" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:20.746852  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc60d8fc83f4cfab23bffcbd1eb72db" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47" is_pre_election: true
I20240503 08:11:20.747092  2820 raft_consensus.cc:2454] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:20.747146  2429 raft_consensus.cc:2454] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 0.
I20240503 08:11:20.747535  2626 leader_election.cc:302] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:20.747802  3688 raft_consensus.cc:2781] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.747939  3688 raft_consensus.cc:482] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.748039  3688 raft_consensus.cc:3037] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.749970  3688 raft_consensus.cc:504] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.750627  3688 leader_election.cc:288] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Requested vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:20.751133  2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc60d8fc83f4cfab23bffcbd1eb72db" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:20.751245  2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc60d8fc83f4cfab23bffcbd1eb72db" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc209b43de14ccbbba8e8770cb02d47"
I20240503 08:11:20.751338  2820 raft_consensus.cc:3037] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.751503  2429 raft_consensus.cc:3037] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.753252  2820 raft_consensus.cc:2454] T bbc60d8fc83f4cfab23bffcbd1eb72db P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:20.753484  2429 raft_consensus.cc:2454] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:20.753757  2626 leader_election.cc:302] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:20.754370  3688 raft_consensus.cc:2781] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:20.754755  3688 raft_consensus.cc:686] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [term 1 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:20.755028  3688 consensus_queue.cc:237] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:20.757830  2282 catalog_manager.cc:5484] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:20.919548  3742 consensus_queue.cc:1035] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:20.939863  3747 consensus_queue.cc:1035] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.114758  3422 consensus_queue.cc:1035] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.129112  3699 consensus_queue.cc:1035] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.163179  3688 consensus_queue.cc:1035] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.178830  3779 consensus_queue.cc:1035] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:21.213379  3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:21.213729  3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:21.222358  3781 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 231.691 ms
  time per row: 0.115846 ms
I20240503 08:11:21.612984  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:45149
--local_ip_for_outbound_sockets=127.2.49.133
--tserver_master_addrs=127.2.49.190:42891
--webserver_port=40491
--webserver_interface=127.2.49.133
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 08:11:21.958699  3840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:21.959129  3840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:21.959312  3840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:21.967937  3840 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:21.968127  3840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:21.968320  3840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:11:21.978564  3840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:45149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=40491
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:21.979991  3840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:21.981782  3840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:21.995719  3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:21.996428  3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:21.997038  3848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:21.998196  3840 server_base.cc:1034] running on GCE node
I20240503 08:11:21.998883  3840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:21.999958  3840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:22.001189  3840 hybrid_clock.cc:648] HybridClock initialized: now 1714723882001145 us; error 36 us; skew 500 ppm
I20240503 08:11:22.001542  3840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:22.009501  3840 webserver.cc:460] Webserver started at http://127.2.49.133:40491/ using document root <none> and password file <none>
I20240503 08:11:22.010440  3840 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:22.010591  3840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:22.016816  3840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 08:11:22.020676  3854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:22.022331  3840 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:11:22.022519  3840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1cc62a1159dd4742b2a609e4756443f2"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-dz87"
I20240503 08:11:22.023396  3840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:22.040585  3840 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:22.041500  3840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:22.041792  3840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:22.042788  3840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:22.044391  3840 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:22.044472  3840 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:22.044601  3840 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:22.044656  3840 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:22.086648  3840 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:45149
I20240503 08:11:22.086763  3966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:45149 every 8 connection(s)
I20240503 08:11:22.088443  3840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:22.092449  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 3840
I20240503 08:11:22.093520  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.134:34697
--local_ip_for_outbound_sockets=127.2.49.134
--tserver_master_addrs=127.2.49.190:42891
--webserver_port=45409
--webserver_interface=127.2.49.134
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:22.118285  3967 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:22.118700  3967 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:22.119796  3967 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:22.121785  2289 ts_manager.cc:194] Re-registered known tserver with Master: 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149)
I20240503 08:11:22.122907  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:33865
W20240503 08:11:22.458137  3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:22.458539  3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:22.458684  3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:22.466948  3971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:22.467097  3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:22.467258  3971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.134
I20240503 08:11:22.476490  3971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.134:34697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.49.134
--webserver_port=45409
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.134
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:22.478022  3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:22.480021  3971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:22.493160  3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.493867  3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.494442  3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:22.495272  3971 server_base.cc:1034] running on GCE node
I20240503 08:11:22.495944  3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:22.496977  3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:22.498144  3971 hybrid_clock.cc:648] HybridClock initialized: now 1714723882498095 us; error 52 us; skew 500 ppm
I20240503 08:11:22.498545  3971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:22.501343  3971 webserver.cc:460] Webserver started at http://127.2.49.134:45409/ using document root <none> and password file <none>
I20240503 08:11:22.502115  3971 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:22.502286  3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:22.509938  3971 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.003s	sys 0.003s
I20240503 08:11:22.513028  3985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:22.514782  3971 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:22.514959  3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2fb0e4ffbda54b9db8c5eceb590719e2"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:22.515794  3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:22.535580  3971 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:22.536492  3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:22.536788  3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:22.537802  3971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:22.539520  3971 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:22.539593  3971 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:22.539700  3971 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:22.539757  3971 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:22.574618  3971 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.134:34697
I20240503 08:11:22.574647  4097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.134:34697 every 8 connection(s)
I20240503 08:11:22.576153  3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:22.582618  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 3971
I20240503 08:11:22.583245  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.135:34325
--local_ip_for_outbound_sockets=127.2.49.135
--tserver_master_addrs=127.2.49.190:42891
--webserver_port=40157
--webserver_interface=127.2.49.135
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:22.593602  4098 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:22.593989  4098 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:22.595094  4098 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:22.597298  2289 ts_manager.cc:194] Re-registered known tserver with Master: 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134:34697)
I20240503 08:11:22.598428  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.134:38621
W20240503 08:11:22.960646  4101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:22.960994  4101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:22.961156  4101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:22.969640  4101 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:22.969800  4101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:22.969978  4101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.135
I20240503 08:11:22.979655  4101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.135:34325
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.49.135
--webserver_port=40157
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.135
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:22.981019  4101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:22.982807  4101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:22.996784  4107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.996786  4108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.997592  4110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:22.998694  4101 server_base.cc:1034] running on GCE node
I20240503 08:11:22.999372  4101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:23.000512  4101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:23.001767  4101 hybrid_clock.cc:648] HybridClock initialized: now 1714723883001716 us; error 41 us; skew 500 ppm
I20240503 08:11:23.002137  4101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:23.004834  4101 webserver.cc:460] Webserver started at http://127.2.49.135:40157/ using document root <none> and password file <none>
I20240503 08:11:23.005738  4101 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:23.005868  4101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:23.012851  4101 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240503 08:11:23.016083  4116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:23.017587  4101 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:11:23.017782  4101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e4a4f8044a424243aa9ab82e0896099d"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-dz87"
I20240503 08:11:23.018815  4101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:23.037011  4101 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:23.037941  4101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:23.038300  4101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:23.039359  4101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:23.041023  4101 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:23.041107  4101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:23.041244  4101 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:23.041299  4101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:23.077354  4101 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.135:34325
I20240503 08:11:23.077414  4228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.135:34325 every 8 connection(s)
I20240503 08:11:23.079238  4101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:23.085777  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4101
I20240503 08:11:23.086593  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.136:34127
--local_ip_for_outbound_sockets=127.2.49.136
--tserver_master_addrs=127.2.49.190:42891
--webserver_port=42877
--webserver_interface=127.2.49.136
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:11:23.098441  4229 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:23.098731  4229 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:23.099794  4229 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:23.101617  2289 ts_manager.cc:194] Re-registered known tserver with Master: e4a4f8044a424243aa9ab82e0896099d (127.2.49.135:34325)
I20240503 08:11:23.102842  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.135:34985
I20240503 08:11:23.126282  3967 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
W20240503 08:11:23.465171  4232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:23.465559  4232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:23.465766  4232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:23.474697  4232 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:23.474869  4232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:23.475031  4232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.136
I20240503 08:11:23.486585  4232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.136:34127
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.49.136
--webserver_port=42877
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.136
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:23.487975  4232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:23.489784  4232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:23.503436  4237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:23.504335  4238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:23.505010  4240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:23.505880  4232 server_base.cc:1034] running on GCE node
I20240503 08:11:23.506738  4232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:23.508533  4232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:23.509802  4232 hybrid_clock.cc:648] HybridClock initialized: now 1714723883509710 us; error 95 us; skew 500 ppm
I20240503 08:11:23.510201  4232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:23.513240  4232 webserver.cc:460] Webserver started at http://127.2.49.136:42877/ using document root <none> and password file <none>
I20240503 08:11:23.514052  4232 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:23.514214  4232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:23.520375  4232 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.002s
I20240503 08:11:23.523913  4246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:23.525477  4232 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:11:23.525666  4232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e8ec7129d89f45bc954e578d526e3eaf"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-dz87"
I20240503 08:11:23.526580  4232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:23.550654  4232 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:23.551684  4232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:23.551991  4232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:23.553288  4232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:23.555022  4232 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:23.555104  4232 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:23.555213  4232 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:23.555269  4232 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:23.602478  4098 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
I20240503 08:11:23.609120  4232 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.136:34127
I20240503 08:11:23.609124  4358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.136:34127 every 8 connection(s)
I20240503 08:11:23.611212  4232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:11:23.613130  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4232
I20240503 08:11:23.632498  4359 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:23.632917  4359 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:23.634235  4359 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:23.636296  2289 ts_manager.cc:194] Re-registered known tserver with Master: e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136:34127)
I20240503 08:11:23.637277  2289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.136:53947
I20240503 08:11:24.106231  4229 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
I20240503 08:11:24.170063  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:24.189915  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:24.198315  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:24.209700  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:24.234887  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:24.239301  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:24.262781  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:24.263882  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:24.641018  4359 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
I20240503 08:11:24.862660  2280 ts_manager.cc:292] Set tserver state for cfc209b43de14ccbbba8e8770cb02d47 to MAINTENANCE_MODE
W20240503 08:11:26.283682  4428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:26.284098  4428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:26.292949  4428 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:26.525089  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:26.525849  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.550839  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.551301  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:26.555717  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:26.563087  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:26.573369  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.597102  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:26.666821  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.688627  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.712095  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.713654  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:26.730398  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:26.762414  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:26.782711  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:26.791879  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.004189  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.009773  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:27.045555  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:27.052147  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.092422  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.107668  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:27.132330  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:27.133416  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:11:27.138214  2365 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:34755, user_credentials={real_user=slave}} blocked reactor thread for 43731.5us
I20240503 08:11:27.401136  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.418349  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:27.429531  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:27.436046  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:27.437048  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.473670  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.474452  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.499358  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:27.595379  2429 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } }
I20240503 08:11:27.604840  2690 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer cfc209b43de14ccbbba8e8770cb02d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:27.605721  4102 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:27.608829  2560 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Refusing update from remote peer cfc209b43de14ccbbba8e8770cb02d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:27.609652  4102 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:27.616035  4553 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } }
I20240503 08:11:27.616932  2690 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } }
I20240503 08:11:27.624066  2280 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 08:11:27.633965  2560 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } }
I20240503 08:11:27.648433  2429 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:27.652559  2690 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer cfc209b43de14ccbbba8e8770cb02d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:27.654485  4553 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:27.655784  2560 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Refusing update from remote peer cfc209b43de14ccbbba8e8770cb02d47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:27.656478  4553 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:27.658414  4553 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:27.660337  2560 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:27.662544  2275 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 390e508ba635499da7e2d5b5ee56e718 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 08:11:27.663426  2363 consensus_peers.cc:485] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 -> Peer e4a4f8044a424243aa9ab82e0896099d (127.2.49.135:34325): Couldn't send request to peer e4a4f8044a424243aa9ab82e0896099d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 390e508ba635499da7e2d5b5ee56e718. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:27.665580  2690 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:27.668963  2282 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 4 to 5, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "cfc209b43de14ccbbba8e8770cb02d47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
W20240503 08:11:27.697090  4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:27.697484  4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:27.706290  4548 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:27.857389  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.857545  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:27.875260  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:27.893263  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.893841  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:27.899410  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:27.915364  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:27.915880  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:28.109716  4652 ts_tablet_manager.cc:902] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Initiating tablet copy from peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:28.116257  4652 tablet_copy_client.cc:308] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:45423
I20240503 08:11:28.124480  2450 tablet_copy_service.cc:140] P cfc209b43de14ccbbba8e8770cb02d47: Received BeginTabletCopySession request for tablet 390e508ba635499da7e2d5b5ee56e718 from peer e4a4f8044a424243aa9ab82e0896099d ({username='slave'} at 127.2.49.135:42659)
I20240503 08:11:28.124845  2450 tablet_copy_service.cc:161] P cfc209b43de14ccbbba8e8770cb02d47: Beginning new tablet copy session on tablet 390e508ba635499da7e2d5b5ee56e718 from peer e4a4f8044a424243aa9ab82e0896099d at {username='slave'} at 127.2.49.135:42659: session id = e4a4f8044a424243aa9ab82e0896099d-390e508ba635499da7e2d5b5ee56e718
I20240503 08:11:28.128751  2450 tablet_copy_source_session.cc:205] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:28.135485  4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e508ba635499da7e2d5b5ee56e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.153613  4652 tablet_copy_client.cc:783] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:28.154073  4652 tablet_copy_client.cc:647] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:28.156689  4656 tablet_copy_client.cc:1002] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Injecting latency into file download: 1500
W20240503 08:11:28.492763  4658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:28.493079  4658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:28.501320  4658 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:28.685815  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:28.701143  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:28.708665  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:28.720741  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:28.733160  2408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:11:28.800976  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:28.824395  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:28.863085  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 08:11:29.019876  4426 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2348
I20240503 08:11:29.028456  2470 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:29.171886  4426 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2348
W20240503 08:11:29.195169  2497 connection.cc:535] server connection from 127.2.49.129:33453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:29.248180  2626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:29.249749  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:29.273365  2756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:29.275696  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:29.284101  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.325728  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.328495  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:11:29.335769  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.343621  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.355275  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.363096  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:29.375290  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:29.388927  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:11:29.443434  2496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:29.445081  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.574954  2800 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:29.588773  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:29.606460  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.620826  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.623325  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.628791  2670 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:29.639189  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:29.657305  4652 tablet_copy_client.cc:523] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:29.662144  4652 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Bootstrap starting.
I20240503 08:11:29.699230  4652 log.cc:824] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Log is configured to *not* fsync() on all Append() calls
W20240503 08:11:29.726236  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.742763  4652 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:29.743355  4652 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Bootstrap complete.
I20240503 08:11:29.743791  4652 ts_tablet_manager.cc:1367] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Time spent bootstrapping tablet: real 0.082s	user 0.054s	sys 0.004s
I20240503 08:11:29.749747  4652 raft_consensus.cc:348] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:29.750221  4652 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4a4f8044a424243aa9ab82e0896099d, State: Initialized, Role: LEARNER
I20240503 08:11:29.750814  4652 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:29.754127  4652 ts_tablet_manager.cc:1398] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.004s
I20240503 08:11:29.756193  2821 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
W20240503 08:11:29.756799  4652 tablet_copy_client.cc:1108] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:29.768029  2690 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:29.769358  2560 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:29.770581  4827 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:29.772351  2755 consensus_peers.cc:485] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 -> Peer 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149): Couldn't send request to peer 1cc62a1159dd4742b2a609e4756443f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36cb470b78894940968683d4bf78bb64. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.774499  4827 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:29.775967  4826 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:29.776597  2560 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:29.782014  2280 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:29.793792  2690 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:29.803655  2560 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } }
I20240503 08:11:29.806619  2690 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:29.806674  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:29.807345  4840 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:29.807675  4554 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:29.810856  4554 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } } }
W20240503 08:11:29.813647  2494 consensus_peers.cc:485] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e -> Peer e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136:34127): Couldn't send request to peer e8ec7129d89f45bc954e578d526e3eaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26168740892c49e2a61445c79ecdb54c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.814684  2690 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } } }
I20240503 08:11:29.816668  2282 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index -1 to 4, NON_VOTER e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:29.827358  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } } }
I20240503 08:11:29.836356  2560 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } }
I20240503 08:11:29.839269  2821 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:29.839470  2690 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:29.839907  4840 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:29.840288  4844 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:29.845435  4843 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
W20240503 08:11:29.846493  2494 consensus_peers.cc:485] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e -> Peer 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134:34697): Couldn't send request to peer 2fb0e4ffbda54b9db8c5eceb590719e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b84ea2b116462ba589ff50802c536f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:29.846206  2821 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
I20240503 08:11:29.846503  2690 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
I20240503 08:11:29.850813  2285 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index -1 to 4, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:30.121343  4815 raft_consensus.cc:482] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.121567  4815 raft_consensus.cc:504] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.122346  4815 leader_election.cc:288] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:30.123700  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
W20240503 08:11:30.124994  2626 leader_election.cc:334] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.125200  2626 leader_election.cc:302] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1443bb164fb4a79a008d9fcf3575b81; no voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.125532  4815 raft_consensus.cc:2726] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:30.128398  4815 raft_consensus.cc:482] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.128583  4815 raft_consensus.cc:504] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.129293  4815 leader_election.cc:288] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:30.130281  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
W20240503 08:11:30.130918  2626 leader_election.cc:334] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.131062  2626 leader_election.cc:302] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1443bb164fb4a79a008d9fcf3575b81; no voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.131331  4815 raft_consensus.cc:2726] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:30.185266  4815 raft_consensus.cc:482] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.185484  4815 raft_consensus.cc:504] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:30.186203  4815 leader_election.cc:288] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:30.186882  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
W20240503 08:11:30.187765  2626 leader_election.cc:334] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.187889  2626 leader_election.cc:302] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1443bb164fb4a79a008d9fcf3575b81; no voters: a15068fdaeae4d89b40ed67c3f37a30e, cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.188100  4815 raft_consensus.cc:2726] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:30.247381  4884 ts_tablet_manager.cc:902] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Initiating tablet copy from peer 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755)
I20240503 08:11:30.249130  4884 tablet_copy_client.cc:308] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:34755
I20240503 08:11:30.256166  2841 tablet_copy_service.cc:140] P 032a50802ce84632bfb57b6a1a840606: Received BeginTabletCopySession request for tablet 36cb470b78894940968683d4bf78bb64 from peer 1cc62a1159dd4742b2a609e4756443f2 ({username='slave'} at 127.2.49.133:54731)
I20240503 08:11:30.256429  2841 tablet_copy_service.cc:161] P 032a50802ce84632bfb57b6a1a840606: Beginning new tablet copy session on tablet 36cb470b78894940968683d4bf78bb64 from peer 1cc62a1159dd4742b2a609e4756443f2 at {username='slave'} at 127.2.49.133:54731: session id = 1cc62a1159dd4742b2a609e4756443f2-36cb470b78894940968683d4bf78bb64
I20240503 08:11:30.259441  2841 tablet_copy_source_session.cc:205] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:30.264269  4890 ts_tablet_manager.cc:902] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Initiating tablet copy from peer a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:30.264755  4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb470b78894940968683d4bf78bb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:30.266323  4890 tablet_copy_client.cc:308] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:40299
I20240503 08:11:30.272970  2581 tablet_copy_service.cc:140] P a15068fdaeae4d89b40ed67c3f37a30e: Received BeginTabletCopySession request for tablet e5b84ea2b116462ba589ff50802c536f from peer 2fb0e4ffbda54b9db8c5eceb590719e2 ({username='slave'} at 127.2.49.134:37749)
I20240503 08:11:30.273269  2581 tablet_copy_service.cc:161] P a15068fdaeae4d89b40ed67c3f37a30e: Beginning new tablet copy session on tablet e5b84ea2b116462ba589ff50802c536f from peer 2fb0e4ffbda54b9db8c5eceb590719e2 at {username='slave'} at 127.2.49.134:37749: session id = 2fb0e4ffbda54b9db8c5eceb590719e2-e5b84ea2b116462ba589ff50802c536f
I20240503 08:11:30.277211  2581 tablet_copy_source_session.cc:205] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:30.277861  4884 tablet_copy_client.cc:783] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:30.278283  4884 tablet_copy_client.cc:647] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:30.280521  4888 tablet_copy_client.cc:1002] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Injecting latency into file download: 1500
I20240503 08:11:30.282359  4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b84ea2b116462ba589ff50802c536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:30.296288  4890 tablet_copy_client.cc:783] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:30.296721  4890 tablet_copy_client.cc:647] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:30.297300  4896 ts_tablet_manager.cc:902] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Initiating tablet copy from peer a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:30.299206  4894 tablet_copy_client.cc:1002] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Injecting latency into file download: 1500
I20240503 08:11:30.299425  4896 tablet_copy_client.cc:308] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:40299
I20240503 08:11:30.300402  4843 raft_consensus.cc:482] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.300659  4843 raft_consensus.cc:504] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:30.301504  4843 leader_election.cc:288] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:30.302093  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
W20240503 08:11:30.303319  2496 leader_election.cc:334] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.303455  2496 leader_election.cc:302] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e; no voters: cfc209b43de14ccbbba8e8770cb02d47, f1443bb164fb4a79a008d9fcf3575b81
I20240503 08:11:30.303820  4554 raft_consensus.cc:2726] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:30.307456  2581 tablet_copy_service.cc:140] P a15068fdaeae4d89b40ed67c3f37a30e: Received BeginTabletCopySession request for tablet 26168740892c49e2a61445c79ecdb54c from peer e8ec7129d89f45bc954e578d526e3eaf ({username='slave'} at 127.2.49.136:55441)
I20240503 08:11:30.307652  2581 tablet_copy_service.cc:161] P a15068fdaeae4d89b40ed67c3f37a30e: Beginning new tablet copy session on tablet 26168740892c49e2a61445c79ecdb54c from peer e8ec7129d89f45bc954e578d526e3eaf at {username='slave'} at 127.2.49.136:55441: session id = e8ec7129d89f45bc954e578d526e3eaf-26168740892c49e2a61445c79ecdb54c
I20240503 08:11:30.309612  2581 tablet_copy_source_session.cc:205] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:30.314800  4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26168740892c49e2a61445c79ecdb54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:30.315429  4554 raft_consensus.cc:482] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.315623  4554 raft_consensus.cc:504] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:30.316309  4554 leader_election.cc:288] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:30.316792  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
W20240503 08:11:30.317816  2496 leader_election.cc:334] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.317936  2496 leader_election.cc:302] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e; no voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.318275  4554 raft_consensus.cc:2726] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:30.326380  4896 tablet_copy_client.cc:783] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:30.326699  4896 tablet_copy_client.cc:647] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:30.328624  4900 tablet_copy_client.cc:1002] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: tablet copy: Injecting latency into file download: 1500
I20240503 08:11:30.332182  4554 raft_consensus.cc:482] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.332446  4554 raft_consensus.cc:504] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:30.333096  4554 leader_election.cc:288] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:30.333616  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "032a50802ce84632bfb57b6a1a840606" is_pre_election: true
I20240503 08:11:30.334223  4826 raft_consensus.cc:482] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.334465  4826 raft_consensus.cc:504] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:30.335155  4826 leader_election.cc:288] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:30.335685  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:30.335897  2560 raft_consensus.cc:2454] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:30.336298  2754 leader_election.cc:302] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
I20240503 08:11:30.336567  4826 raft_consensus.cc:2781] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:30.336648  4826 raft_consensus.cc:482] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.336756  4826 raft_consensus.cc:3037] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:11:30.337066  2756 leader_election.cc:334] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:30.356227  2496 leader_election.cc:334] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.356393  2496 leader_election.cc:302] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e; no voters: 032a50802ce84632bfb57b6a1a840606, cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.356671  4554 raft_consensus.cc:2726] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:30.394291  4826 raft_consensus.cc:504] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:30.395049  4826 leader_election.cc:288] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:30.395758  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd324aef02344416b643a2421ca46918" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:30.396005  2560 raft_consensus.cc:3037] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:11:30.396641  2756 leader_election.cc:334] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.398543  2560 raft_consensus.cc:2454] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 2.
I20240503 08:11:30.398983  2754 leader_election.cc:302] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.399303  4826 raft_consensus.cc:2781] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:30.399716  4826 raft_consensus.cc:686] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [term 2 LEADER]: Becoming Leader. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:30.400008  4826 consensus_queue.cc:237] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:30.403309  2286 catalog_manager.cc:5484] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 reported cstate change: term changed from 1 to 2, leader changed from cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) to 032a50802ce84632bfb57b6a1a840606 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:30.469528  4826 raft_consensus.cc:482] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.469735  4826 raft_consensus.cc:504] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:30.470450  4826 leader_election.cc:288] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:30.471403  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:30.471611  2560 raft_consensus.cc:2454] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:30.472040  2754 leader_election.cc:302] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: 
W20240503 08:11:30.472239  2756 leader_election.cc:334] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.472319  4826 raft_consensus.cc:2781] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:30.472442  4826 raft_consensus.cc:482] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.472539  4826 raft_consensus.cc:3037] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.492020  4917 raft_consensus.cc:482] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.492229  4917 raft_consensus.cc:504] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.492990  4917 leader_election.cc:288] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:30.493957  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:30.494189  2690 raft_consensus.cc:2454] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
W20240503 08:11:30.494578  2756 leader_election.cc:334] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.494859  2756 leader_election.cc:302] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, f1443bb164fb4a79a008d9fcf3575b81; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.495151  4917 raft_consensus.cc:2781] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:30.495278  4917 raft_consensus.cc:482] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.495376  4917 raft_consensus.cc:3037] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.523221  4571 raft_consensus.cc:482] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.523432  4571 raft_consensus.cc:504] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:30.524245  4571 leader_election.cc:288] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:30.524844  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" is_pre_election: true
I20240503 08:11:30.525135  2560 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 1.
I20240503 08:11:30.525627  2624 leader_election.cc:302] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:30.525959  4571 raft_consensus.cc:2781] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 08:11:30.525959  2626 leader_election.cc:334] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.526127  4571 raft_consensus.cc:482] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.526255  4571 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.526374  4922 raft_consensus.cc:482] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.526567  4922 raft_consensus.cc:504] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.527542  4922 leader_election.cc:288] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:30.528362  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:30.528556  2690 raft_consensus.cc:2454] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 1.
I20240503 08:11:30.528931  2756 leader_election.cc:302] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:30.529186  4922 raft_consensus.cc:2781] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:30.529297  4922 raft_consensus.cc:482] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.529387  4922 raft_consensus.cc:3037] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:11:30.529748  2756 leader_election.cc:334] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.553624  4826 raft_consensus.cc:504] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:30.553630  4571 raft_consensus.cc:504] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:30.554677  4826 leader_election.cc:288] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:30.555153  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:30.555373  2560 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.556174  4917 raft_consensus.cc:504] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.556895  4917 leader_election.cc:288] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
W20240503 08:11:30.557807  2756 leader_election.cc:334] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.558231  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22756ba18d2f49999f02ac878b8bf03d" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:30.558485  2690 raft_consensus.cc:3037] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.558604  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62c007e061d4a96b4a3c8633bd29ee5" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:30.558809  2561 raft_consensus.cc:3037] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:11:30.558907  2756 leader_election.cc:334] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.559273  4571 leader_election.cc:288] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
W20240503 08:11:30.560322  2626 leader_election.cc:334] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.659552  2690 raft_consensus.cc:2454] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 2.
I20240503 08:11:30.659552  2560 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
I20240503 08:11:30.659561  4922 raft_consensus.cc:504] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.660329  2624 leader_election.cc:302] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.660501  4922 leader_election.cc:288] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:30.660954  4571 raft_consensus.cc:2781] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:30.661105  4571 raft_consensus.cc:686] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:30.661458  2756 leader_election.cc:302] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, f1443bb164fb4a79a008d9fcf3575b81; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.661408  4571 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:30.662039  4922 raft_consensus.cc:2781] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:30.662212  4922 raft_consensus.cc:686] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [term 2 LEADER]: Becoming Leader. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:30.662544  4922 consensus_queue.cc:237] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
W20240503 08:11:30.664166  2756 leader_election.cc:334] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.664121  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9906f233cd4c848eb58f2330ca70b3" candidate_uuid: "032a50802ce84632bfb57b6a1a840606" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:30.664348  2690 raft_consensus.cc:3037] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.666558  2286 catalog_manager.cc:5484] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 reported cstate change: term changed from 1 to 2, leader changed from cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) to 032a50802ce84632bfb57b6a1a840606 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:30.667529  2282 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 1 to 2, leader changed from cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:30.672508  4554 raft_consensus.cc:482] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.672710  4554 raft_consensus.cc:504] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:30.673341  4554 leader_election.cc:288] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:30.674294  2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81" is_pre_election: true
I20240503 08:11:30.674618  2691 raft_consensus.cc:2454] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 1.
W20240503 08:11:30.674788  2496 leader_election.cc:334] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.675036  2496 leader_election.cc:302] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.675328  4554 raft_consensus.cc:2781] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:30.675437  4554 raft_consensus.cc:482] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cfc209b43de14ccbbba8e8770cb02d47)
I20240503 08:11:30.675521  4554 raft_consensus.cc:3037] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:30.753803  2561 raft_consensus.cc:2454] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 2.
I20240503 08:11:30.753780  2690 raft_consensus.cc:2454] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032a50802ce84632bfb57b6a1a840606 in term 2.
I20240503 08:11:30.754611  2756 leader_election.cc:302] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, f1443bb164fb4a79a008d9fcf3575b81; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.755767  2754 leader_election.cc:302] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.756553  4938 raft_consensus.cc:2781] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:30.756714  4938 raft_consensus.cc:686] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [term 2 LEADER]: Becoming Leader. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:30.756834  4826 raft_consensus.cc:2781] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:30.757001  4826 raft_consensus.cc:686] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [term 2 LEADER]: Becoming Leader. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:30.757144  4938 consensus_queue.cc:237] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:30.757336  4826 consensus_queue.cc:237] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:30.763131  2282 catalog_manager.cc:5484] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 reported cstate change: term changed from 1 to 2, leader changed from cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) to 032a50802ce84632bfb57b6a1a840606 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:30.763656  2282 catalog_manager.cc:5484] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 reported cstate change: term changed from 1 to 2, leader changed from cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) to 032a50802ce84632bfb57b6a1a840606 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:30.836450  2561 raft_consensus.cc:1261] T cd324aef02344416b643a2421ca46918 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:11:30.837134  4938 consensus_queue.cc:1035] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:30.841998  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:30.852523  4554 raft_consensus.cc:504] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:30.853384  4554 leader_election.cc:288] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 election: Requested vote from peers cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:30.854104  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5268a78550334d9b828888bc358ea695" candidate_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:30.854383  2690 raft_consensus.cc:3037] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:11:30.857512  2496 leader_election.cc:334] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:30.954270  2690 raft_consensus.cc:2454] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15068fdaeae4d89b40ed67c3f37a30e in term 2.
I20240503 08:11:30.954869  2496 leader_election.cc:302] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: cfc209b43de14ccbbba8e8770cb02d47
I20240503 08:11:30.955314  4554 raft_consensus.cc:2781] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:30.955508  4554 raft_consensus.cc:686] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 LEADER]: Becoming Leader. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Running, Role: LEADER
I20240503 08:11:30.955785  4554 consensus_queue.cc:237] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:30.958863  2286 catalog_manager.cc:5484] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: term changed from 1 to 2, leader changed from cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) to a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130). New cstate: current_term: 2 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:31.070747  4184 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Advancing to term 2
I20240503 08:11:31.103121  2690 raft_consensus.cc:1261] T 22756ba18d2f49999f02ac878b8bf03d P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:11:31.103787  4922 consensus_queue.cc:1035] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:31.110450  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:31.129034  2561 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:31.129658  4571 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:11:31.134609  2626 consensus_peers.cc:485] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:31.153733  4184 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 LEARNER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:31.154886  4571 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:31.159191  4571 raft_consensus.cc:1050] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: attempting to promote NON_VOTER e4a4f8044a424243aa9ab82e0896099d to VOTER
I20240503 08:11:31.160620  4571 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:31.162892  4184 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 LEARNER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:11:31.163403  2561 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:11:31.163517  4815 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:11:31.163959  2626 consensus_peers.cc:485] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:31.164325  4571 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:31.166683  4571 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.167331  2561 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.170986  4184 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.171862  2286 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 5 to 7, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.178469  2690 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:31.180361  2561 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:11:31.180464  4184 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:11:31.181034  4815 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:31.181289  4988 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:31.183837  4571 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.184602  2561 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.186492  4184 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:31.188328  2275 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 390e508ba635499da7e2d5b5ee56e718 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:11:31.190999  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:31.191817  2285 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 7 to 8, VOTER cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:11:31.197901  2275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:31.199477  2275 catalog_manager.cc:4573] TS cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 390e508ba635499da7e2d5b5ee56e718: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:31.209332  2561 raft_consensus.cc:1261] T a62c007e061d4a96b4a3c8633bd29ee5 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:11:31.210002  4938 consensus_queue.cc:1035] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:31.308050  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:31.319701  2690 raft_consensus.cc:1261] T 1b9906f233cd4c848eb58f2330ca70b3 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:11:31.320420  4826 consensus_queue.cc:1035] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:31.473322  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:31.490775  2690 raft_consensus.cc:1261] T 5268a78550334d9b828888bc358ea695 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:11:31.491492  4843 consensus_queue.cc:1035] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:31.497390  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:31.583578  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:11:31.685120  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:31.781250  4884 tablet_copy_client.cc:523] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:31.784919  4884 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap starting.
W20240503 08:11:31.796671  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:31.797211  4884 log.cc:824] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:31.799789  4890 tablet_copy_client.cc:523] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:31.804234  4890 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Bootstrap starting.
W20240503 08:11:31.813022  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:31.817504  4890 log.cc:824] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:31.829265  4896 tablet_copy_client.cc:523] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:31.835256  4896 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Bootstrap starting.
I20240503 08:11:31.860900  4896 log.cc:824] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:31.872721  4884 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:31.873471  4884 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap complete.
I20240503 08:11:31.875250  4884 ts_tablet_manager.cc:1367] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Time spent bootstrapping tablet: real 0.091s	user 0.054s	sys 0.012s
I20240503 08:11:31.884215  4884 raft_consensus.cc:348] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:31.884980  4884 raft_consensus.cc:729] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc62a1159dd4742b2a609e4756443f2, State: Initialized, Role: LEARNER
I20240503 08:11:31.885671  4884 consensus_queue.cc:260] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:31.890878  4890 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:31.891721  4890 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Bootstrap complete.
I20240503 08:11:31.892431  4890 ts_tablet_manager.cc:1367] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Time spent bootstrapping tablet: real 0.088s	user 0.070s	sys 0.004s
I20240503 08:11:31.894932  4884 ts_tablet_manager.cc:1398] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.008s
I20240503 08:11:31.899612  4890 raft_consensus.cc:348] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } }
I20240503 08:11:31.900278  4890 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2fb0e4ffbda54b9db8c5eceb590719e2, State: Initialized, Role: LEARNER
I20240503 08:11:31.900964  4890 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } }
I20240503 08:11:31.901417  2841 tablet_copy_service.cc:342] P 032a50802ce84632bfb57b6a1a840606: Request end of tablet copy session 1cc62a1159dd4742b2a609e4756443f2-36cb470b78894940968683d4bf78bb64 received from {username='slave'} at 127.2.49.133:54731
I20240503 08:11:31.901602  2841 tablet_copy_service.cc:433] P 032a50802ce84632bfb57b6a1a840606: ending tablet copy session 1cc62a1159dd4742b2a609e4756443f2-36cb470b78894940968683d4bf78bb64 on tablet 36cb470b78894940968683d4bf78bb64 with peer 1cc62a1159dd4742b2a609e4756443f2
I20240503 08:11:31.909750  4890 ts_tablet_manager.cc:1398] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.006s
I20240503 08:11:31.910815  2581 tablet_copy_service.cc:342] P a15068fdaeae4d89b40ed67c3f37a30e: Request end of tablet copy session 2fb0e4ffbda54b9db8c5eceb590719e2-e5b84ea2b116462ba589ff50802c536f received from {username='slave'} at 127.2.49.134:37749
I20240503 08:11:31.910991  2581 tablet_copy_service.cc:433] P a15068fdaeae4d89b40ed67c3f37a30e: ending tablet copy session 2fb0e4ffbda54b9db8c5eceb590719e2-e5b84ea2b116462ba589ff50802c536f on tablet e5b84ea2b116462ba589ff50802c536f with peer 2fb0e4ffbda54b9db8c5eceb590719e2
I20240503 08:11:31.944875  4896 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:31.945611  4896 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Bootstrap complete.
I20240503 08:11:31.946223  4896 ts_tablet_manager.cc:1367] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Time spent bootstrapping tablet: real 0.111s	user 0.087s	sys 0.011s
I20240503 08:11:31.954577  4896 raft_consensus.cc:348] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } }
I20240503 08:11:31.955217  4896 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8ec7129d89f45bc954e578d526e3eaf, State: Initialized, Role: LEARNER
I20240503 08:11:31.955921  4896 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: NON_VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: true } }
I20240503 08:11:31.958021  4896 ts_tablet_manager.cc:1398] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.001s
I20240503 08:11:31.959209  2581 tablet_copy_service.cc:342] P a15068fdaeae4d89b40ed67c3f37a30e: Request end of tablet copy session e8ec7129d89f45bc954e578d526e3eaf-26168740892c49e2a61445c79ecdb54c received from {username='slave'} at 127.2.49.136:55441
I20240503 08:11:31.959355  2581 tablet_copy_service.cc:433] P a15068fdaeae4d89b40ed67c3f37a30e: ending tablet copy session e8ec7129d89f45bc954e578d526e3eaf-26168740892c49e2a61445c79ecdb54c on tablet 26168740892c49e2a61445c79ecdb54c with peer e8ec7129d89f45bc954e578d526e3eaf
W20240503 08:11:32.018345  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:32.077917  4314 raft_consensus.cc:1203] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:11:32.100867  3922 raft_consensus.cc:1203] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:11:32.386873  4053 raft_consensus.cc:1203] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 08:11:32.413975  2275 catalog_manager.cc:4573] TS cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 390e508ba635499da7e2d5b5ee56e718: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:32.542516  5078 raft_consensus.cc:1050] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: attempting to promote NON_VOTER e8ec7129d89f45bc954e578d526e3eaf to VOTER
I20240503 08:11:32.543368  5078 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } }
I20240503 08:11:32.546245  5080 raft_consensus.cc:1050] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: attempting to promote NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 to VOTER
I20240503 08:11:32.547106  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 LEARNER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.547588  2690 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.547431  5080 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:32.548077  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.547976  5077 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.548521  5078 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.548942  5077 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:11:32.550087  2561 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.550660  2690 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.550772  3922 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 1 LEARNER]: Refusing update from remote peer 032a50802ce84632bfb57b6a1a840606: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.551271  5080 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.551719  4917 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.553668  5081 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.558277  5077 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.559374  5080 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:32.560863  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.561142  3922 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:32.566702  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.567131  2282 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 reported cstate change: config changed from index 4 to 5, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "032a50802ce84632bfb57b6a1a840606" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:32.568985  2690 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:32.569504  2286 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 4 to 5, e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:32.570749  2561 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:32.576472  2690 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.618005  2561 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } }
I20240503 08:11:32.621068  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:11:32.621989  5077 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:32.623678  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:11:32.626076  5077 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.627079  5079 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:32.629338  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.629591  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } }
I20240503 08:11:32.630379  2273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26168740892c49e2a61445c79ecdb54c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:32.632685  2282 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 5 to 6, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:32.637444  2670 tablet_service.cc:1514] Processing DeleteTablet for tablet 26168740892c49e2a61445c79ecdb54c with delete_type TABLET_DATA_TOMBSTONED (TS f1443bb164fb4a79a008d9fcf3575b81 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48878
I20240503 08:11:32.638469  5106 tablet_replica.cc:330] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: stopping tablet replica
I20240503 08:11:32.639071  5106 raft_consensus.cc:2229] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:32.639451  5106 raft_consensus.cc:2258] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:32.641170  5106 ts_tablet_manager.cc:1875] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:32.647404  5106 ts_tablet_manager.cc:1888] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:11:32.647644  5106 log.cc:1196] T 26168740892c49e2a61445c79ecdb54c P f1443bb164fb4a79a008d9fcf3575b81: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/26168740892c49e2a61445c79ecdb54c
I20240503 08:11:32.648610  2275 catalog_manager.cc:4830] TS f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461): tablet 26168740892c49e2a61445c79ecdb54c (table rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]) successfully deleted
I20240503 08:11:32.674510  2821 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36cb470b78894940968683d4bf78bb64"
dest_uuid: "032a50802ce84632bfb57b6a1a840606"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33310
I20240503 08:11:32.674901  2821 raft_consensus.cc:595] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:32.712702  2821 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36cb470b78894940968683d4bf78bb64"
dest_uuid: "032a50802ce84632bfb57b6a1a840606"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33322
I20240503 08:11:32.712904  2821 raft_consensus.cc:595] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:32.870473  5079 raft_consensus.cc:1050] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: attempting to promote NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 to VOTER
I20240503 08:11:32.871191  5079 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:32.873761  2821 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.873858  2690 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.873999  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 1 LEARNER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:32.874346  5079 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.874650  4554 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.875008  5077 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:32.878465  4554 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:32.879432  2690 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:32.879966  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:32.883199  2821 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:32.884596  2289 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 4 to 5, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:32.933907  2561 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f"
dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49122
I20240503 08:11:32.934227  2561 raft_consensus.cc:595] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:32.952690  2821 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36cb470b78894940968683d4bf78bb64"
dest_uuid: "032a50802ce84632bfb57b6a1a840606"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33346
I20240503 08:11:32.952904  2821 raft_consensus.cc:595] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:33.003813  5081 raft_consensus.cc:979] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: : Instructing follower f1443bb164fb4a79a008d9fcf3575b81 to start an election
I20240503 08:11:33.004033  5081 raft_consensus.cc:1067] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Signalling peer f1443bb164fb4a79a008d9fcf3575b81 to start an election
I20240503 08:11:33.005084  2690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36cb470b78894940968683d4bf78bb64"
dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
 from {username='slave'} at 127.2.49.132:34617
I20240503 08:11:33.005347  2690 raft_consensus.cc:482] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:33.005478  2690 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.007529  2690 raft_consensus.cc:504] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:33.008525  2690 leader_election.cc:288] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: Requested vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149)
I20240503 08:11:33.008872  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:33.008984  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:33.009109  2821 raft_consensus.cc:3032] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:33.009224  2821 raft_consensus.cc:729] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 032a50802ce84632bfb57b6a1a840606, State: Running, Role: LEADER
I20240503 08:11:33.009215  2561 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.009557  2821 consensus_queue.cc:260] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:33.010080  2821 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.011937  2561 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
I20240503 08:11:33.012254  2821 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
I20240503 08:11:33.012712  2626 leader_election.cc:302] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:33.013034  4571 raft_consensus.cc:2781] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:33.013428  4571 raft_consensus.cc:686] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:33.013792  4571 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:33.016713  2286 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 1 to 2, leader changed from 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:33.016978  3922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb470b78894940968683d4bf78bb64" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1cc62a1159dd4742b2a609e4756443f2"
I20240503 08:11:33.017305  3922 raft_consensus.cc:3037] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.020591  3922 raft_consensus.cc:2454] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
I20240503 08:11:33.172199  2561 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f"
dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49130
I20240503 08:11:33.172427  2561 raft_consensus.cc:595] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:33.324859  5079 raft_consensus.cc:979] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: : Instructing follower f1443bb164fb4a79a008d9fcf3575b81 to start an election
I20240503 08:11:33.325030  5079 raft_consensus.cc:1067] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Signalling peer f1443bb164fb4a79a008d9fcf3575b81 to start an election
I20240503 08:11:33.325938  2690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f"
dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
 from {username='slave'} at 127.2.49.130:53239
I20240503 08:11:33.326143  2690 raft_consensus.cc:482] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:33.326319  2690 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.328332  2690 raft_consensus.cc:504] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:33.329209  2690 leader_election.cc:288] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: Requested vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134:34697)
I20240503 08:11:33.329747  2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:33.329970  2561 raft_consensus.cc:3032] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:33.330076  2561 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Running, Role: LEADER
I20240503 08:11:33.330029  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:33.330262  2821 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.330715  2561 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:33.331230  2561 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.333146  2821 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
I20240503 08:11:33.334455  2561 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
I20240503 08:11:33.335124  2624 leader_election.cc:302] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:33.335422  4571 raft_consensus.cc:2781] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:33.335610  4571 raft_consensus.cc:686] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Becoming Leader. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:33.335968  4571 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:33.336547  4053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "f1443bb164fb4a79a008d9fcf3575b81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2"
I20240503 08:11:33.336882  4053 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:33.338835  2286 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: term changed from 1 to 2, leader changed from a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) to f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:33.339071  4053 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1443bb164fb4a79a008d9fcf3575b81 in term 2.
W20240503 08:11:33.418097  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:11:33.482558  2275 catalog_manager.cc:4573] TS cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 390e508ba635499da7e2d5b5ee56e718: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:33.513352  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:33.564479  3922 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:33.565264  5157 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:33.571772  2561 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:33.572410  2821 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:33.572392  5157 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:33.573161  5157 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:33.605533  2690 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:33.608243  2561 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:11:33.608880  5192 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:33.615612  3921 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:11:33.618103  5192 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:33.619645  4571 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:33.621052  2561 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:33.625157  2275 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36cb470b78894940968683d4bf78bb64 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:11:33.625780  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:33.626782  2286 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 5 to 7, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:33.626222  3921 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:33.632222  2800 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cb470b78894940968683d4bf78bb64 with delete_type TABLET_DATA_TOMBSTONED (TS 032a50802ce84632bfb57b6a1a840606 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53080
I20240503 08:11:33.640996  5197 tablet_replica.cc:330] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: stopping tablet replica
I20240503 08:11:33.641542  5197 raft_consensus.cc:2229] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:33.641844  5197 raft_consensus.cc:2258] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:33.644371  5197 ts_tablet_manager.cc:1875] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:33.650413  5197 ts_tablet_manager.cc:1888] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:33.650580  5197 log.cc:1196] T 36cb470b78894940968683d4bf78bb64 P 032a50802ce84632bfb57b6a1a840606: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/36cb470b78894940968683d4bf78bb64
I20240503 08:11:33.651648  2275 catalog_manager.cc:4830] TS 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755): tablet 36cb470b78894940968683d4bf78bb64 (table rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]) successfully deleted
W20240503 08:11:33.653105  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:11:33.785599  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:11:33.803822  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:33.804463  4571 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:33.853219  2561 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:33.853797  4571 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:33.857405  2821 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:33.858332  5157 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:11:33.891814  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:34.091867  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:11:34.263464  2626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:11:34.264422  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:34.374434  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:34.382717  2756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 46 similar messages]
I20240503 08:11:34.411412  2690 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:34.413583  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:11:34.413584  2821 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:11:34.414310  5212 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:34.414705  5230 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:34.417567  5157 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:34.418268  2821 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:34.418433  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:34.424324  2282 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 reported cstate change: config changed from index 5 to 7, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:34.425036  2275 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5b84ea2b116462ba589ff50802c536f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:34.434198  2539 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b84ea2b116462ba589ff50802c536f with delete_type TABLET_DATA_TOMBSTONED (TS a15068fdaeae4d89b40ed67c3f37a30e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45246
I20240503 08:11:34.438752  5235 tablet_replica.cc:330] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: stopping tablet replica
I20240503 08:11:34.439203  5235 raft_consensus.cc:2229] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:34.439437  5235 raft_consensus.cc:2258] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:34.440975  5235 ts_tablet_manager.cc:1875] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:34.446087  5235 ts_tablet_manager.cc:1888] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:34.446298  5235 log.cc:1196] T e5b84ea2b116462ba589ff50802c536f P a15068fdaeae4d89b40ed67c3f37a30e: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5b84ea2b116462ba589ff50802c536f
I20240503 08:11:34.447263  2273 catalog_manager.cc:4830] TS a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299): tablet e5b84ea2b116462ba589ff50802c536f (table rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]) successfully deleted
W20240503 08:11:34.500698  2496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:11:34.551719  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:34.561642  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:11:34.730710  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:34.732316  2800 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:34.742965  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:34.760670  4164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:34.804982  2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:34.810907  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:34.834300  2539 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:34.913983  2690 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } }
I20240503 08:11:34.916551  2560 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:11:34.916745  3921 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:11:34.917435  5157 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:11:34.917814  2624 consensus_peers.cc:485] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134:34697): Couldn't send request to peer 2fb0e4ffbda54b9db8c5eceb590719e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36cb470b78894940968683d4bf78bb64. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:34.918092  5212 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:34.921268  5212 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
I20240503 08:11:34.921854  2560 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
I20240503 08:11:34.922603  3921 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
I20240503 08:11:34.927625  2289 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 7 to 8, NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) added. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } } }
I20240503 08:11:34.948143  2690 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:34.950522  2560 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:34.950731  4184 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:34.951076  5157 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:11:34.951432  2625 consensus_peers.cc:485] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149): Couldn't send request to peer 1cc62a1159dd4742b2a609e4756443f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 390e508ba635499da7e2d5b5ee56e718. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:34.951588  5212 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:34.953998  5157 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:34.954672  2560 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:34.954946  4184 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:34.959198  2286 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 8 to 9, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:34.973122  2690 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:34.975334  2821 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:11:34.975492  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 08:11:34.975559  2624 consensus_peers.cc:485] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 -> Peer e4a4f8044a424243aa9ab82e0896099d (127.2.49.135:34325): Couldn't send request to peer e4a4f8044a424243aa9ab82e0896099d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b84ea2b116462ba589ff50802c536f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:34.976006  5212 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:34.976363  5230 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:34.978824  5157 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:34.979560  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:34.979518  2821 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:34.984190  2282 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 reported cstate change: config changed from index 7 to 8, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:35.466877  5338 ts_tablet_manager.cc:902] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: Initiating tablet copy from peer f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:35.470551  5338 tablet_copy_client.cc:308] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:38461
I20240503 08:11:35.478513  2711 tablet_copy_service.cc:140] P f1443bb164fb4a79a008d9fcf3575b81: Received BeginTabletCopySession request for tablet 390e508ba635499da7e2d5b5ee56e718 from peer 1cc62a1159dd4742b2a609e4756443f2 ({username='slave'} at 127.2.49.133:50487)
I20240503 08:11:35.478745  2711 tablet_copy_service.cc:161] P f1443bb164fb4a79a008d9fcf3575b81: Beginning new tablet copy session on tablet 390e508ba635499da7e2d5b5ee56e718 from peer 1cc62a1159dd4742b2a609e4756443f2 at {username='slave'} at 127.2.49.133:50487: session id = 1cc62a1159dd4742b2a609e4756443f2-390e508ba635499da7e2d5b5ee56e718
I20240503 08:11:35.481318  2711 tablet_copy_source_session.cc:205] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:35.483419  5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e508ba635499da7e2d5b5ee56e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.488945  5338 tablet_copy_client.cc:783] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:35.489252  5338 tablet_copy_client.cc:647] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:35.491830  5347 tablet_copy_client.cc:1002] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Injecting latency into file download: 1500
I20240503 08:11:35.516944  5355 ts_tablet_manager.cc:902] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: Initiating tablet copy from peer f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:35.518628  5355 tablet_copy_client.cc:308] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:38461
I20240503 08:11:35.526010  2711 tablet_copy_service.cc:140] P f1443bb164fb4a79a008d9fcf3575b81: Received BeginTabletCopySession request for tablet 36cb470b78894940968683d4bf78bb64 from peer 2fb0e4ffbda54b9db8c5eceb590719e2 ({username='slave'} at 127.2.49.134:58273)
I20240503 08:11:35.526285  2711 tablet_copy_service.cc:161] P f1443bb164fb4a79a008d9fcf3575b81: Beginning new tablet copy session on tablet 36cb470b78894940968683d4bf78bb64 from peer 2fb0e4ffbda54b9db8c5eceb590719e2 at {username='slave'} at 127.2.49.134:58273: session id = 2fb0e4ffbda54b9db8c5eceb590719e2-36cb470b78894940968683d4bf78bb64
I20240503 08:11:35.528553  2711 tablet_copy_source_session.cc:205] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:35.530042  5355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb470b78894940968683d4bf78bb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.535032  5355 tablet_copy_client.cc:783] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:35.535307  5355 tablet_copy_client.cc:647] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:35.536921  5359 tablet_copy_client.cc:1002] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Injecting latency into file download: 1500
W20240503 08:11:35.583241  2275 catalog_manager.cc:4573] TS cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 390e508ba635499da7e2d5b5ee56e718: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:35.586540  5363 ts_tablet_manager.cc:902] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: Initiating tablet copy from peer f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:35.592499  5363 tablet_copy_client.cc:308] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:38461
I20240503 08:11:35.598776  2711 tablet_copy_service.cc:140] P f1443bb164fb4a79a008d9fcf3575b81: Received BeginTabletCopySession request for tablet e5b84ea2b116462ba589ff50802c536f from peer e4a4f8044a424243aa9ab82e0896099d ({username='slave'} at 127.2.49.135:38529)
I20240503 08:11:35.598994  2711 tablet_copy_service.cc:161] P f1443bb164fb4a79a008d9fcf3575b81: Beginning new tablet copy session on tablet e5b84ea2b116462ba589ff50802c536f from peer e4a4f8044a424243aa9ab82e0896099d at {username='slave'} at 127.2.49.135:38529: session id = e4a4f8044a424243aa9ab82e0896099d-e5b84ea2b116462ba589ff50802c536f
I20240503 08:11:35.602134  2711 tablet_copy_source_session.cc:205] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:35.603705  5363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b84ea2b116462ba589ff50802c536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.608357  5363 tablet_copy_client.cc:783] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:35.608642  5363 tablet_copy_client.cc:647] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:35.610284  5367 tablet_copy_client.cc:1002] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Injecting latency into file download: 1500
W20240503 08:11:36.041049  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:36.055804  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:36.141435  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:36.211032  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:36.310756  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:36.348933  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:11:36.551796  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:36.716097  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:36.736690  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:36.932531  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:11:36.992266  5338 tablet_copy_client.cc:523] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:36.995126  5338 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap starting.
I20240503 08:11:37.031728  5338 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:37.032159  5338 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap complete.
I20240503 08:11:37.032425  5338 ts_tablet_manager.cc:1367] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: Time spent bootstrapping tablet: real 0.037s	user 0.032s	sys 0.004s
I20240503 08:11:37.033257  5338 raft_consensus.cc:348] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:37.033483  5338 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc62a1159dd4742b2a609e4756443f2, State: Initialized, Role: LEARNER
I20240503 08:11:37.033640  5338 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:37.034411  5338 ts_tablet_manager.cc:1398] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.001s
I20240503 08:11:37.035441  2711 tablet_copy_service.cc:342] P f1443bb164fb4a79a008d9fcf3575b81: Request end of tablet copy session 1cc62a1159dd4742b2a609e4756443f2-390e508ba635499da7e2d5b5ee56e718 received from {username='slave'} at 127.2.49.133:50487
I20240503 08:11:37.035616  2711 tablet_copy_service.cc:433] P f1443bb164fb4a79a008d9fcf3575b81: ending tablet copy session 1cc62a1159dd4742b2a609e4756443f2-390e508ba635499da7e2d5b5ee56e718 on tablet 390e508ba635499da7e2d5b5ee56e718 with peer 1cc62a1159dd4742b2a609e4756443f2
I20240503 08:11:37.037321  5355 tablet_copy_client.cc:523] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:37.041249  5355 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: Bootstrap starting.
I20240503 08:11:37.077288  5355 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:37.078531  5355 tablet_bootstrap.cc:492] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: Bootstrap complete.
I20240503 08:11:37.078903  5355 ts_tablet_manager.cc:1367] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: Time spent bootstrapping tablet: real 0.038s	user 0.030s	sys 0.009s
I20240503 08:11:37.080089  5355 raft_consensus.cc:348] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } }
I20240503 08:11:37.080399  5355 raft_consensus.cc:729] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2fb0e4ffbda54b9db8c5eceb590719e2, State: Initialized, Role: LEARNER
I20240503 08:11:37.080655  5355 consensus_queue.cc:260] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: NON_VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: true } }
I20240503 08:11:37.081586  5355 ts_tablet_manager.cc:1398] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:11:37.083549  2711 tablet_copy_service.cc:342] P f1443bb164fb4a79a008d9fcf3575b81: Request end of tablet copy session 2fb0e4ffbda54b9db8c5eceb590719e2-36cb470b78894940968683d4bf78bb64 received from {username='slave'} at 127.2.49.134:58273
I20240503 08:11:37.083711  2711 tablet_copy_service.cc:433] P f1443bb164fb4a79a008d9fcf3575b81: ending tablet copy session 2fb0e4ffbda54b9db8c5eceb590719e2-36cb470b78894940968683d4bf78bb64 on tablet 36cb470b78894940968683d4bf78bb64 with peer 2fb0e4ffbda54b9db8c5eceb590719e2
I20240503 08:11:37.100785  3921 raft_consensus.cc:1203] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 08:11:37.110715  5363 tablet_copy_client.cc:523] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:37.114001  5363 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: Bootstrap starting.
I20240503 08:11:37.153048  5363 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:37.153489  5363 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: Bootstrap complete.
I20240503 08:11:37.153753  5363 ts_tablet_manager.cc:1367] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: Time spent bootstrapping tablet: real 0.040s	user 0.036s	sys 0.003s
I20240503 08:11:37.154636  5363 raft_consensus.cc:348] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:37.154871  5363 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4a4f8044a424243aa9ab82e0896099d, State: Initialized, Role: LEARNER
I20240503 08:11:37.155095  5363 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:37.155897  5363 ts_tablet_manager.cc:1398] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240503 08:11:37.156791  2711 tablet_copy_service.cc:342] P f1443bb164fb4a79a008d9fcf3575b81: Request end of tablet copy session e4a4f8044a424243aa9ab82e0896099d-e5b84ea2b116462ba589ff50802c536f received from {username='slave'} at 127.2.49.135:38529
I20240503 08:11:37.156958  2711 tablet_copy_service.cc:433] P f1443bb164fb4a79a008d9fcf3575b81: ending tablet copy session e4a4f8044a424243aa9ab82e0896099d-e5b84ea2b116462ba589ff50802c536f on tablet e5b84ea2b116462ba589ff50802c536f with peer e4a4f8044a424243aa9ab82e0896099d
W20240503 08:11:37.269497  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:11:37.303848  4053 raft_consensus.cc:1203] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:11:37.589439  5509 raft_consensus.cc:1050] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: attempting to promote NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 to VOTER
I20240503 08:11:37.590471  5509 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:37.599905  3921 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:37.600700  4184 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:37.600682  5508 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:37.601606  5508 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:37.603461  2560 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:37.604641  5509 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 08:11:37.605361  5509 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:37.606752  4184 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:37.610721  3921 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:37.613831  2286 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d reported cstate change: config changed from index 9 to 10, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:37.616919  2561 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:37.680223  4184 raft_consensus.cc:1203] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:11:37.827322  5509 raft_consensus.cc:1050] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81: attempting to promote NON_VOTER 2fb0e4ffbda54b9db8c5eceb590719e2 to VOTER
I20240503 08:11:37.828238  5509 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:37.830583  4053 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 LEARNER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:37.830808  2560 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:37.831264  5509 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:37.831468  3921 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:37.831980  2690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718"
dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45750
I20240503 08:11:37.832077  5514 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:37.832247  2690 raft_consensus.cc:595] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:11:37.833155  5514 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:37.836196  5509 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.837342  2560 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.841436  4053 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.845871  2280 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 8 to 9, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:37.848954  3921 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.858255  2690 consensus_queue.cc:237] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } }
I20240503 08:11:37.860592  3921 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:37.860612  4053 raft_consensus.cc:1261] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:37.861213  5509 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:37.861572  5514 consensus_queue.cc:1035] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:37.864398  5508 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.865146  4053 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.865483  3921 raft_consensus.cc:2932] T 36cb470b78894940968683d4bf78bb64 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } }
I20240503 08:11:37.868737  2275 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36cb470b78894940968683d4bf78bb64 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:37.869840  2289 catalog_manager.cc:5484] T 36cb470b78894940968683d4bf78bb64 P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 9 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:37.873988  2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cb470b78894940968683d4bf78bb64 with delete_type TABLET_DATA_TOMBSTONED (TS a15068fdaeae4d89b40ed67c3f37a30e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45246
I20240503 08:11:37.883297  5549 tablet_replica.cc:330] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: stopping tablet replica
I20240503 08:11:37.883626  5549 raft_consensus.cc:2229] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:37.884011  5549 raft_consensus.cc:2258] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:37.885605  5549 ts_tablet_manager.cc:1875] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:37.891060  5549 ts_tablet_manager.cc:1888] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:11:37.891211  5549 log.cc:1196] T 36cb470b78894940968683d4bf78bb64 P a15068fdaeae4d89b40ed67c3f37a30e: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/36cb470b78894940968683d4bf78bb64
I20240503 08:11:37.892154  2273 catalog_manager.cc:4830] TS a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299): tablet 36cb470b78894940968683d4bf78bb64 (table rebalance_test_table_2 [id=927e8a8760074c68b08e76002c5d5c87]) successfully deleted
I20240503 08:11:37.903445  2690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718"
dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45768
I20240503 08:11:37.903687  2690 raft_consensus.cc:595] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:11:38.033509  5508 raft_consensus.cc:979] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: : Instructing follower e4a4f8044a424243aa9ab82e0896099d to start an election
I20240503 08:11:38.033747  5212 raft_consensus.cc:1067] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Signalling peer e4a4f8044a424243aa9ab82e0896099d to start an election
I20240503 08:11:38.034832  4184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718"
dest_uuid: "e4a4f8044a424243aa9ab82e0896099d"
 from {username='slave'} at 127.2.49.131:53611
I20240503 08:11:38.035168  4184 raft_consensus.cc:482] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:38.035312  4184 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:38.037245  4184 raft_consensus.cc:504] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:38.039047  4184 leader_election.cc:288] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [CANDIDATE]: Term 3 election: Requested vote from peers a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149)
I20240503 08:11:38.039757  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:38.040014  2690 raft_consensus.cc:3032] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:11:38.040122  2690 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:38.040524  2690 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:38.041075  2690 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:38.045243  2690 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 3.
I20240503 08:11:38.054597  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:38.054867  2560 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:38.057232  2560 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 3.
I20240503 08:11:38.057626  3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e508ba635499da7e2d5b5ee56e718" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "1cc62a1159dd4742b2a609e4756443f2"
I20240503 08:11:38.057852  3921 raft_consensus.cc:3037] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:38.057776  4117 leader_election.cc:302] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a15068fdaeae4d89b40ed67c3f37a30e, e4a4f8044a424243aa9ab82e0896099d, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:38.058256  5480 raft_consensus.cc:2781] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:11:38.058871  5480 raft_consensus.cc:686] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 3 LEADER]: Becoming Leader. State: Replica: e4a4f8044a424243aa9ab82e0896099d, State: Running, Role: LEADER
I20240503 08:11:38.059471  5480 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:38.060918  3921 raft_consensus.cc:2454] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 3.
I20240503 08:11:38.063812  2280 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d reported cstate change: term changed from 2 to 3, leader changed from f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) to e4a4f8044a424243aa9ab82e0896099d (127.2.49.135). New cstate: current_term: 3 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:38.106956  5212 raft_consensus.cc:1050] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: attempting to promote NON_VOTER e4a4f8044a424243aa9ab82e0896099d to VOTER
I20240503 08:11:38.107951  5212 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:38.111542  4184 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 LEARNER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:38.111811  2821 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:38.112272  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:38.112236  5509 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:38.112859  5508 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:38.113950  5212 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:38.123540  5509 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.124702  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.125056  2821 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.129666  4184 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.130393  2280 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 8 to 9, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:38.137202  2690 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:38.142722  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:38.143359  5509 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:38.147261  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.146541  5508 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.153147  2275 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5b84ea2b116462ba589ff50802c536f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:38.153177  4184 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:38.154284  5508 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:38.156109  2280 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 9 to 10, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:38.156829  4184 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:38.166780  2800 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b84ea2b116462ba589ff50802c536f with delete_type TABLET_DATA_TOMBSTONED (TS 032a50802ce84632bfb57b6a1a840606 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53080
I20240503 08:11:38.167603  5586 tablet_replica.cc:330] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: stopping tablet replica
I20240503 08:11:38.167832  5586 raft_consensus.cc:2229] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:38.168298  5586 raft_consensus.cc:2258] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:38.169234  5586 ts_tablet_manager.cc:1875] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:38.175587  5586 ts_tablet_manager.cc:1888] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:11:38.175753  5586 log.cc:1196] T e5b84ea2b116462ba589ff50802c536f P 032a50802ce84632bfb57b6a1a840606: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5b84ea2b116462ba589ff50802c536f
I20240503 08:11:38.176725  2275 catalog_manager.cc:4830] TS 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755): tablet e5b84ea2b116462ba589ff50802c536f (table rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]) successfully deleted
I20240503 08:11:38.410950  2560 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:38.414343  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:11:38.414636  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:11:38.415187  5603 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:38.415627  5079 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:11:38.419238  5079 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:38.420292  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
I20240503 08:11:38.421520  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } } }
W20240503 08:11:38.421929  2494 consensus_peers.cc:485] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e -> Peer e4a4f8044a424243aa9ab82e0896099d (127.2.49.135:34325): Couldn't send request to peer e4a4f8044a424243aa9ab82e0896099d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26168740892c49e2a61445c79ecdb54c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:38.425493  2282 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e reported cstate change: config changed from index 6 to 7, NON_VOTER e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:38.495571  3921 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:11:38.496310  5480 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:38.506517  2560 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:11:38.507176  5480 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:38.520713  2690 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:11:38.521482  5569 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:11:38.534266  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:11:38.539510  4184 consensus_queue.cc:237] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:38.541636  2560 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:11:38.542265  3921 raft_consensus.cc:1261] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:11:38.542320  5624 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:38.542769  5620 consensus_queue.cc:1035] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:38.544840  5480 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:38.545853  2560 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P a15068fdaeae4d89b40ed67c3f37a30e [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:38.545825  3921 raft_consensus.cc:2932] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:38.551301  2273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 390e508ba635499da7e2d5b5ee56e718 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:38.551054  2280 catalog_manager.cc:5484] T 390e508ba635499da7e2d5b5ee56e718 P 1cc62a1159dd4742b2a609e4756443f2 reported cstate change: config changed from index 10 to 12, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New cstate: current_term: 3 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:38.555291  2670 tablet_service.cc:1514] Processing DeleteTablet for tablet 390e508ba635499da7e2d5b5ee56e718 with delete_type TABLET_DATA_TOMBSTONED (TS f1443bb164fb4a79a008d9fcf3575b81 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:48878
I20240503 08:11:38.556106  5626 tablet_replica.cc:330] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: stopping tablet replica
I20240503 08:11:38.556319  5626 raft_consensus.cc:2229] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:38.556555  5626 raft_consensus.cc:2258] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:38.557601  5626 ts_tablet_manager.cc:1875] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:38.562865  5626 ts_tablet_manager.cc:1888] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 08:11:38.563005  5626 log.cc:1196] T 390e508ba635499da7e2d5b5ee56e718 P f1443bb164fb4a79a008d9fcf3575b81: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/390e508ba635499da7e2d5b5ee56e718
I20240503 08:11:38.563763  2275 catalog_manager.cc:4830] TS f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461): tablet 390e508ba635499da7e2d5b5ee56e718 (table rebalance_test_table_1 [id=06c43ae20c4a4579acfe7c7d339ffe52]) successfully deleted
W20240503 08:11:38.701493  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:38.719101  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:38.761245  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:38.937196  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:11:38.946444  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:11:38.952508  5649 ts_tablet_manager.cc:902] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Initiating tablet copy from peer a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299)
I20240503 08:11:38.954293  5649 tablet_copy_client.cc:308] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:40299
I20240503 08:11:38.955032  2581 tablet_copy_service.cc:140] P a15068fdaeae4d89b40ed67c3f37a30e: Received BeginTabletCopySession request for tablet 26168740892c49e2a61445c79ecdb54c from peer e4a4f8044a424243aa9ab82e0896099d ({username='slave'} at 127.2.49.135:56237)
I20240503 08:11:38.955171  2581 tablet_copy_service.cc:161] P a15068fdaeae4d89b40ed67c3f37a30e: Beginning new tablet copy session on tablet 26168740892c49e2a61445c79ecdb54c from peer e4a4f8044a424243aa9ab82e0896099d at {username='slave'} at 127.2.49.135:56237: session id = e4a4f8044a424243aa9ab82e0896099d-26168740892c49e2a61445c79ecdb54c
I20240503 08:11:38.957121  2581 tablet_copy_source_session.cc:205] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:38.958483  5649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26168740892c49e2a61445c79ecdb54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:38.963384  5649 tablet_copy_client.cc:783] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:38.963652  5649 tablet_copy_client.cc:647] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:38.965328  5653 tablet_copy_client.cc:1002] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Injecting latency into file download: 1500
W20240503 08:11:39.111541  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:39.157205  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:39.270308  2626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:11:39.271332  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:39.356328  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:39.398677  2756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:11:39.531754  2496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:11:39.698117  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:39.702584  2275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 08:11:39.703900  2275 catalog_manager.cc:4573] TS cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 390e508ba635499da7e2d5b5ee56e718: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
I20240503 08:11:40.465840  5649 tablet_copy_client.cc:523] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:40.468607  5649 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Bootstrap starting.
I20240503 08:11:40.504696  5649 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:40.505178  5649 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Bootstrap complete.
I20240503 08:11:40.505468  5649 ts_tablet_manager.cc:1367] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Time spent bootstrapping tablet: real 0.037s	user 0.028s	sys 0.008s
I20240503 08:11:40.506276  5649 raft_consensus.cc:348] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:40.506554  5649 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4a4f8044a424243aa9ab82e0896099d, State: Initialized, Role: LEARNER
I20240503 08:11:40.506742  5649 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:40.507297  5649 ts_tablet_manager.cc:1398] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:11:40.508239  2581 tablet_copy_service.cc:342] P a15068fdaeae4d89b40ed67c3f37a30e: Request end of tablet copy session e4a4f8044a424243aa9ab82e0896099d-26168740892c49e2a61445c79ecdb54c received from {username='slave'} at 127.2.49.135:56237
I20240503 08:11:40.508411  2581 tablet_copy_service.cc:433] P a15068fdaeae4d89b40ed67c3f37a30e: ending tablet copy session e4a4f8044a424243aa9ab82e0896099d-26168740892c49e2a61445c79ecdb54c on tablet 26168740892c49e2a61445c79ecdb54c with peer e4a4f8044a424243aa9ab82e0896099d
I20240503 08:11:40.519614  4184 raft_consensus.cc:1203] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 08:11:40.933709  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:11:40.938423  5680 raft_consensus.cc:1050] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: attempting to promote NON_VOTER e4a4f8044a424243aa9ab82e0896099d to VOTER
I20240503 08:11:40.939177  5680 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:40.942099  4184 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 LEARNER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:11:40.942350  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:11:40.942587  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Refusing update from remote peer a15068fdaeae4d89b40ed67c3f37a30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:11:40.942977  5710 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:40.943393  5708 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:40.944056  5708 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:11:40.949386  5708 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:40.950098  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:40.955660  4184 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:40.956879  2280 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 reported cstate change: config changed from index 7 to 8, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:40.958499  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4a4f8044a424243aa9ab82e0896099d (127.2.49.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:41.014724  2560 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c"
dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49288
I20240503 08:11:41.014961  2560 raft_consensus.cc:595] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Received request to transfer leadership
W20240503 08:11:41.207496  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:41.228377  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:11:41.235942  2560 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c"
dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49308
I20240503 08:11:41.236162  2560 raft_consensus.cc:595] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Received request to transfer leadership
W20240503 08:11:41.311874  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:11:41.393244  5680 raft_consensus.cc:979] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: : Instructing follower e4a4f8044a424243aa9ab82e0896099d to start an election
I20240503 08:11:41.393502  5711 raft_consensus.cc:1067] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Signalling peer e4a4f8044a424243aa9ab82e0896099d to start an election
I20240503 08:11:41.394351  4184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c"
dest_uuid: "e4a4f8044a424243aa9ab82e0896099d"
 from {username='slave'} at 127.2.49.130:51891
I20240503 08:11:41.394578  4184 raft_consensus.cc:482] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:41.394726  4184 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:41.396929  4184 raft_consensus.cc:504] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:41.397889  4184 leader_election.cc:288] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [CANDIDATE]: Term 2 election: Requested vote from peers 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755), a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299), e8ec7129d89f45bc954e578d526e3eaf (127.2.49.136:34127)
I20240503 08:11:41.398864  2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a15068fdaeae4d89b40ed67c3f37a30e"
I20240503 08:11:41.399179  2560 raft_consensus.cc:3032] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:41.399284  2560 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a15068fdaeae4d89b40ed67c3f37a30e, State: Running, Role: LEADER
I20240503 08:11:41.399713  2560 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:41.400375  2560 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:41.405093  2560 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 2.
I20240503 08:11:41.412130  2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "032a50802ce84632bfb57b6a1a840606"
I20240503 08:11:41.412384  2821 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:41.413373  4314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26168740892c49e2a61445c79ecdb54c" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e8ec7129d89f45bc954e578d526e3eaf"
I20240503 08:11:41.413686  4314 raft_consensus.cc:3037] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:41.414752  2821 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 2.
I20240503 08:11:41.415268  4119 leader_election.cc:302] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 032a50802ce84632bfb57b6a1a840606, a15068fdaeae4d89b40ed67c3f37a30e, e4a4f8044a424243aa9ab82e0896099d; no voters: 
I20240503 08:11:41.415638  5686 raft_consensus.cc:2781] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:41.416055  5686 raft_consensus.cc:686] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 LEADER]: Becoming Leader. State: Replica: e4a4f8044a424243aa9ab82e0896099d, State: Running, Role: LEADER
I20240503 08:11:41.416438  5686 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:41.416849  4314 raft_consensus.cc:2454] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 2.
I20240503 08:11:41.419564  2280 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d reported cstate change: term changed from 1 to 2, leader changed from a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) to e4a4f8044a424243aa9ab82e0896099d (127.2.49.135). New cstate: current_term: 2 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:11:41.490742  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:41.538035  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:41.541669  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:11:41.634491  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:11:41.799558  2560 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:41.800225  5686 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:11:41.844094  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:11:41.864041  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:41.864826  5686 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:11:41.870790  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:11:41.871448  5686 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:11:41.944375  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:42.312011  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:11:42.388576  4184 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } }
I20240503 08:11:42.390537  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:42.390664  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:11:42.391146  5777 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:42.391475  5778 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:11:42.393872  5777 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:42.394671  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:42.394965  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } }
I20240503 08:11:42.398267  2273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26168740892c49e2a61445c79ecdb54c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:42.400040  2282 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d reported cstate change: config changed from index 8 to 10, VOTER a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.405346  2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 26168740892c49e2a61445c79ecdb54c with delete_type TABLET_DATA_TOMBSTONED (TS a15068fdaeae4d89b40ed67c3f37a30e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45246
I20240503 08:11:42.406080  5783 tablet_replica.cc:330] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: stopping tablet replica
I20240503 08:11:42.406332  5783 raft_consensus.cc:2229] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:42.406562  5783 raft_consensus.cc:2258] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:42.407439  5783 ts_tablet_manager.cc:1875] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:42.412482  5783 ts_tablet_manager.cc:1888] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:11:42.412617  5783 log.cc:1196] T 26168740892c49e2a61445c79ecdb54c P a15068fdaeae4d89b40ed67c3f37a30e: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/26168740892c49e2a61445c79ecdb54c
I20240503 08:11:42.413376  2273 catalog_manager.cc:4830] TS a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299): tablet 26168740892c49e2a61445c79ecdb54c (table rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]) successfully deleted
I20240503 08:11:42.657212  2800 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:42.672946  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:42.674683  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:42.711910  2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:42.712422  4164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:42.740002  2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:11:42.748834  2754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:40299, user_credentials={real_user=slave}} blocked reactor thread for 78141.7us
I20240503 08:11:42.763334  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:42.828336  4184 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:42.830688  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:11:42.830880  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240503 08:11:42.831275  4118 consensus_peers.cc:485] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d -> Peer 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149): Couldn't send request to peer 1cc62a1159dd4742b2a609e4756443f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26168740892c49e2a61445c79ecdb54c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:42.831470  5777 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:42.831802  5763 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:42.834872  5763 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:42.835902  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:42.838785  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:42.841086  2285 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d reported cstate change: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New cstate: current_term: 2 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:43.212044  5845 ts_tablet_manager.cc:902] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: Initiating tablet copy from peer e4a4f8044a424243aa9ab82e0896099d (127.2.49.135:34325)
I20240503 08:11:43.213431  5845 tablet_copy_client.cc:308] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.135:34325
I20240503 08:11:43.220263  4204 tablet_copy_service.cc:140] P e4a4f8044a424243aa9ab82e0896099d: Received BeginTabletCopySession request for tablet 26168740892c49e2a61445c79ecdb54c from peer 1cc62a1159dd4742b2a609e4756443f2 ({username='slave'} at 127.2.49.133:53179)
I20240503 08:11:43.220548  4204 tablet_copy_service.cc:161] P e4a4f8044a424243aa9ab82e0896099d: Beginning new tablet copy session on tablet 26168740892c49e2a61445c79ecdb54c from peer 1cc62a1159dd4742b2a609e4756443f2 at {username='slave'} at 127.2.49.133:53179: session id = 1cc62a1159dd4742b2a609e4756443f2-26168740892c49e2a61445c79ecdb54c
I20240503 08:11:43.223515  4204 tablet_copy_source_session.cc:205] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:43.225322  5845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26168740892c49e2a61445c79ecdb54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:43.230702  5845 tablet_copy_client.cc:783] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:43.230988  5845 tablet_copy_client.cc:647] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:43.233114  5849 tablet_copy_client.cc:1002] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Injecting latency into file download: 1500
W20240503 08:11:43.325472  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:43.548040  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:43.796736  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:43.891058  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:43.937484  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:44.061997  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:44.073619  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:11:44.340016  2626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:11:44.340868  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:44.343557  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:44.527629  2756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:11:44.539141  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:44.666492  2496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:11:44.736747  5845 tablet_copy_client.cc:523] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:44.739504  5845 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap starting.
I20240503 08:11:44.778317  5845 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:44.778717  5845 tablet_bootstrap.cc:492] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap complete.
I20240503 08:11:44.778980  5845 ts_tablet_manager.cc:1367] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: Time spent bootstrapping tablet: real 0.040s	user 0.031s	sys 0.008s
I20240503 08:11:44.779609  5845 raft_consensus.cc:348] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:44.779837  5845 raft_consensus.cc:729] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc62a1159dd4742b2a609e4756443f2, State: Initialized, Role: LEARNER
I20240503 08:11:44.780017  5845 consensus_queue.cc:260] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:44.780704  5845 ts_tablet_manager.cc:1398] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 08:11:44.781669  4204 tablet_copy_service.cc:342] P e4a4f8044a424243aa9ab82e0896099d: Request end of tablet copy session 1cc62a1159dd4742b2a609e4756443f2-26168740892c49e2a61445c79ecdb54c received from {username='slave'} at 127.2.49.133:53179
I20240503 08:11:44.781844  4204 tablet_copy_service.cc:433] P e4a4f8044a424243aa9ab82e0896099d: ending tablet copy session 1cc62a1159dd4742b2a609e4756443f2-26168740892c49e2a61445c79ecdb54c on tablet 26168740892c49e2a61445c79ecdb54c with peer 1cc62a1159dd4742b2a609e4756443f2
W20240503 08:11:45.113152  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:11:45.202831  3921 raft_consensus.cc:1203] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11]   Dedup: 2.11->[]
I20240503 08:11:45.639021  5919 raft_consensus.cc:1050] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d: attempting to promote NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 to VOTER
I20240503 08:11:45.639895  5919 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:45.642436  3921 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:11:45.642565  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:11:45.642612  2821 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:11:45.643129  5919 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:45.643474  5918 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:45.643786  5921 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:45.647356  5918 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.648283  3921 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.651741  2821 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.652499  2280 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d reported cstate change: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:45.662393  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } attrs { replace: true } } peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.669796  4184 consensus_queue.cc:237] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:45.671528  4314 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:11:45.671547  3921 raft_consensus.cc:1261] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:11:45.672058  5919 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:11:45.672358  5921 consensus_queue.cc:1035] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:11:45.674585  5878 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.675232  3921 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.675391  4314 raft_consensus.cc:2932] T 26168740892c49e2a61445c79ecdb54c P e8ec7129d89f45bc954e578d526e3eaf [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:45.678259  2273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26168740892c49e2a61445c79ecdb54c with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:45.680313  2286 catalog_manager.cc:5484] T 26168740892c49e2a61445c79ecdb54c P e4a4f8044a424243aa9ab82e0896099d reported cstate change: config changed from index 12 to 13, VOTER 032a50802ce84632bfb57b6a1a840606 (127.2.49.132) evicted. New cstate: current_term: 2 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e8ec7129d89f45bc954e578d526e3eaf" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 34127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:45.683838  2800 tablet_service.cc:1514] Processing DeleteTablet for tablet 26168740892c49e2a61445c79ecdb54c with delete_type TABLET_DATA_TOMBSTONED (TS 032a50802ce84632bfb57b6a1a840606 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:53080
I20240503 08:11:45.684530  5930 tablet_replica.cc:330] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: stopping tablet replica
I20240503 08:11:45.684741  5930 raft_consensus.cc:2229] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:45.685086  5930 raft_consensus.cc:2258] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:45.685972  5930 ts_tablet_manager.cc:1875] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:45.690428  5930 ts_tablet_manager.cc:1888] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 08:11:45.690577  5930 log.cc:1196] T 26168740892c49e2a61445c79ecdb54c P 032a50802ce84632bfb57b6a1a840606: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/26168740892c49e2a61445c79ecdb54c
I20240503 08:11:45.691322  2275 catalog_manager.cc:4830] TS 032a50802ce84632bfb57b6a1a840606 (127.2.49.132:34755): tablet 26168740892c49e2a61445c79ecdb54c (table rebalance_test_table_3 [id=bebb0f4aa1de4c19810801614c60fff3]) successfully deleted
W20240503 08:11:45.692905  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:11:45.824959  2690 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:45.827656  4184 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240503 08:11:45.827785  2625 consensus_peers.cc:485] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 -> Peer 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149): Couldn't send request to peer 1cc62a1159dd4742b2a609e4756443f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b84ea2b116462ba589ff50802c536f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:45.828254  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:11:45.828258  5642 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:45.828811  5642 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:45.831980  5943 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:45.832947  4184 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
I20240503 08:11:45.837446  2280 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:45.846873  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } } }
W20240503 08:11:46.132148  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:46.225170  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:46.277751  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:11:46.447129  5963 ts_tablet_manager.cc:902] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: Initiating tablet copy from peer f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461)
I20240503 08:11:46.448958  5963 tablet_copy_client.cc:308] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:38461
I20240503 08:11:46.449784  2711 tablet_copy_service.cc:140] P f1443bb164fb4a79a008d9fcf3575b81: Received BeginTabletCopySession request for tablet e5b84ea2b116462ba589ff50802c536f from peer 1cc62a1159dd4742b2a609e4756443f2 ({username='slave'} at 127.2.49.133:50487)
I20240503 08:11:46.449965  2711 tablet_copy_service.cc:161] P f1443bb164fb4a79a008d9fcf3575b81: Beginning new tablet copy session on tablet e5b84ea2b116462ba589ff50802c536f from peer 1cc62a1159dd4742b2a609e4756443f2 at {username='slave'} at 127.2.49.133:50487: session id = 1cc62a1159dd4742b2a609e4756443f2-e5b84ea2b116462ba589ff50802c536f
I20240503 08:11:46.452087  2711 tablet_copy_source_session.cc:205] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:46.453361  5963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b84ea2b116462ba589ff50802c536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:46.458045  5963 tablet_copy_client.cc:783] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:46.458405  5963 tablet_copy_client.cc:647] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:46.460108  5967 tablet_copy_client.cc:1002] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Injecting latency into file download: 1500
W20240503 08:11:46.517088  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:46.611708  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:11:46.767501  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:46.852035  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:46.867081  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:46.944264  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:47.909932  2275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:47.911258  2275 catalog_manager.cc:4573] TS cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 390e508ba635499da7e2d5b5ee56e718: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111)
W20240503 08:11:47.944996  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:11:47.960609  5963 tablet_copy_client.cc:523] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:47.964568  5963 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap starting.
I20240503 08:11:48.001158  5963 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:48.001590  5963 tablet_bootstrap.cc:492] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: Bootstrap complete.
I20240503 08:11:48.001856  5963 ts_tablet_manager.cc:1367] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: Time spent bootstrapping tablet: real 0.037s	user 0.032s	sys 0.004s
I20240503 08:11:48.002543  5963 raft_consensus.cc:348] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:48.002791  5963 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc62a1159dd4742b2a609e4756443f2, State: Initialized, Role: LEARNER
I20240503 08:11:48.003026  5963 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: true } }
I20240503 08:11:48.003749  3921 raft_consensus.cc:1203] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11]   Dedup: 2.11->[]
I20240503 08:11:48.003892  5963 ts_tablet_manager.cc:1398] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240503 08:11:48.004729  2711 tablet_copy_service.cc:342] P f1443bb164fb4a79a008d9fcf3575b81: Request end of tablet copy session 1cc62a1159dd4742b2a609e4756443f2-e5b84ea2b116462ba589ff50802c536f received from {username='slave'} at 127.2.49.133:50487
I20240503 08:11:48.004848  2711 tablet_copy_service.cc:433] P f1443bb164fb4a79a008d9fcf3575b81: ending tablet copy session 1cc62a1159dd4742b2a609e4756443f2-e5b84ea2b116462ba589ff50802c536f on tablet e5b84ea2b116462ba589ff50802c536f with peer 1cc62a1159dd4742b2a609e4756443f2
W20240503 08:11:48.218129  2496 consensus_peers.cc:485] T 5268a78550334d9b828888bc358ea695 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:11:48.499681  6029 raft_consensus.cc:1050] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: attempting to promote NON_VOTER 1cc62a1159dd4742b2a609e4756443f2 to VOTER
I20240503 08:11:48.500655  6029 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:48.513434  3921 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 2 LEARNER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:11:48.513434  4184 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:11:48.513720  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Refusing update from remote peer f1443bb164fb4a79a008d9fcf3575b81: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:11:48.514343  6031 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:48.514827  6029 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:48.515216  5985 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:11:48.519040  5985 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:48.519667  4053 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:48.519722  3921 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:48.524335  4184 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:48.525565  2280 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 reported cstate change: config changed from index 11 to 12, 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f1443bb164fb4a79a008d9fcf3575b81" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:48.610777  2690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f"
dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54834
I20240503 08:11:48.610998  2690 raft_consensus.cc:595] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Received request to transfer leadership
W20240503 08:11:48.668787  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:11:48.832150  2690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f"
dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54852
I20240503 08:11:48.832424  2690 raft_consensus.cc:595] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Received request to transfer leadership
W20240503 08:11:48.908802  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:48.922149  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:48.997980  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:11:49.028445  6061 raft_consensus.cc:979] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: : Instructing follower e4a4f8044a424243aa9ab82e0896099d to start an election
I20240503 08:11:49.028653  6029 raft_consensus.cc:1067] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Signalling peer e4a4f8044a424243aa9ab82e0896099d to start an election
I20240503 08:11:49.029381  4184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f"
dest_uuid: "e4a4f8044a424243aa9ab82e0896099d"
 from {username='slave'} at 127.2.49.131:53611
I20240503 08:11:49.029577  4184 raft_consensus.cc:482] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:49.029717  4184 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:49.031772  4184 raft_consensus.cc:504] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:49.033175  2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "f1443bb164fb4a79a008d9fcf3575b81"
I20240503 08:11:49.033211  3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "1cc62a1159dd4742b2a609e4756443f2"
I20240503 08:11:49.033326  4184 leader_election.cc:288] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [CANDIDATE]: Term 3 election: Requested vote from peers f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461), 2fb0e4ffbda54b9db8c5eceb590719e2 (127.2.49.134:34697), 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149)
I20240503 08:11:49.033435  2690 raft_consensus.cc:3032] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:11:49.033461  3921 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:49.033538  2690 raft_consensus.cc:729] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f1443bb164fb4a79a008d9fcf3575b81, State: Running, Role: LEADER
I20240503 08:11:49.033843  2690 consensus_queue.cc:260] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:49.034351  2690 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:49.037065  2690 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 3.
I20240503 08:11:49.037204  3921 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 3.
I20240503 08:11:49.037837  4118 leader_election.cc:302] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cc62a1159dd4742b2a609e4756443f2, e4a4f8044a424243aa9ab82e0896099d, f1443bb164fb4a79a008d9fcf3575b81; no voters: 
I20240503 08:11:49.038259  5921 raft_consensus.cc:2781] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:11:49.038805  5921 raft_consensus.cc:686] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 3 LEADER]: Becoming Leader. State: Replica: e4a4f8044a424243aa9ab82e0896099d, State: Running, Role: LEADER
I20240503 08:11:49.039181  5921 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:49.043663  4053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b84ea2b116462ba589ff50802c536f" candidate_uuid: "e4a4f8044a424243aa9ab82e0896099d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2"
I20240503 08:11:49.043134  2286 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d reported cstate change: term changed from 2 to 3, leader changed from f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) to e4a4f8044a424243aa9ab82e0896099d (127.2.49.135). New cstate: current_term: 3 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:49.044162  4053 raft_consensus.cc:3037] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:49.049402  4053 raft_consensus.cc:2454] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4a4f8044a424243aa9ab82e0896099d in term 3.
W20240503 08:11:49.345783  2626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:11:49.346804  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:11:49.366639  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:11:49.404210  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:11:49.442344  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:11:49.492307  3921 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:11:49.493201  5921 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:11:49.515908  4053 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:11:49.516574  5921 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:11:49.521375  2690 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:11:49.521978  5921 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:11:49.526201  4184 consensus_queue.cc:237] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } }
I20240503 08:11:49.529074  3921 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 08:11:49.529860  6087 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:11:49.530436  4052 raft_consensus.cc:1261] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 3 FOLLOWER]: Refusing update from remote peer e4a4f8044a424243aa9ab82e0896099d: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 08:11:49.531510  6087 consensus_queue.cc:1035] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:11:49.532593  6087 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P e4a4f8044a424243aa9ab82e0896099d [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:49.533035  3921 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:49.535048  4052 raft_consensus.cc:2932] T e5b84ea2b116462ba589ff50802c536f P 2fb0e4ffbda54b9db8c5eceb590719e2 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:49.538321  2273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5b84ea2b116462ba589ff50802c536f with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:49.538553  2285 catalog_manager.cc:5484] T e5b84ea2b116462ba589ff50802c536f P 1cc62a1159dd4742b2a609e4756443f2 reported cstate change: config changed from index 12 to 14, VOTER f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131) evicted. New cstate: current_term: 3 leader_uuid: "e4a4f8044a424243aa9ab82e0896099d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2fb0e4ffbda54b9db8c5eceb590719e2" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 34697 } attrs { promote: false } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: false } } peers { permanent_uuid: "1cc62a1159dd4742b2a609e4756443f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 45149 } attrs { promote: false } } }
I20240503 08:11:49.542774  2670 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b84ea2b116462ba589ff50802c536f with delete_type TABLET_DATA_TOMBSTONED (TS f1443bb164fb4a79a008d9fcf3575b81 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:48878
I20240503 08:11:49.543403  6091 tablet_replica.cc:330] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: stopping tablet replica
I20240503 08:11:49.543608  6091 raft_consensus.cc:2229] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.543793  6091 raft_consensus.cc:2258] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.544631  6091 ts_tablet_manager.cc:1875] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.549235  6091 ts_tablet_manager.cc:1888] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240503 08:11:49.549350  6091 log.cc:1196] T e5b84ea2b116462ba589ff50802c536f P f1443bb164fb4a79a008d9fcf3575b81: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/e5b84ea2b116462ba589ff50802c536f
I20240503 08:11:49.550068  2275 catalog_manager.cc:4830] TS f1443bb164fb4a79a008d9fcf3575b81 (127.2.49.131:38461): tablet e5b84ea2b116462ba589ff50802c536f (table rebalance_test_table_0 [id=33353c9c8f2040a3b1884a44f670ba73]) successfully deleted
W20240503 08:11:49.560176  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:11:49.580109  2756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:11:49.805521  2800 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:49.841411  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:49.845747  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:49.853407  2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:49.866946  2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:49.876091  4164 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:49.904244  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:11:49.910697  2496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:11:50.065671  2800 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:50.071061  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:50.072691  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:50.091730  4164 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:50.119240  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:50.123515  2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.133929  2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:50.388240  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:45423
--local_ip_for_outbound_sockets=127.2.49.129
--tserver_master_addrs=127.2.49.190:42891
--webserver_port=43747
--webserver_interface=127.2.49.129
--builtin_ntp_servers=127.2.49.148:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 08:11:50.636898  2496 consensus_peers.cc:485] T e9aa470143b74329b6f63d3ff1b1ec86 P a15068fdaeae4d89b40ed67c3f37a30e -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:11:50.762041  6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:50.762391  6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:50.762552  6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:50.770318  6167 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:50.770444  6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:50.770606  6167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:11:50.780009  6167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:45423
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=43747
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.2.49.190:42891
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:50.781347  6167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:50.783135  6167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:50.795686  6172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:50.796365  6173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:50.797035  6175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:50.797883  6167 server_base.cc:1034] running on GCE node
I20240503 08:11:50.798617  6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:50.799638  6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:50.800848  6167 hybrid_clock.cc:648] HybridClock initialized: now 1714723910800790 us; error 56 us; skew 500 ppm
I20240503 08:11:50.801183  6167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:50.803568  6167 webserver.cc:460] Webserver started at http://127.2.49.129:43747/ using document root <none> and password file <none>
I20240503 08:11:50.804323  6167 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:50.804453  6167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:50.809691  6167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:11:50.813787  6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:50.815284  6167 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240503 08:11:50.815451  6167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cfc209b43de14ccbbba8e8770cb02d47"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dz87"
I20240503 08:11:50.816279  6167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:50.846390  6167 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:50.847258  6167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:50.847539  6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:50.848534  6167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:50.850968  6188 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240503 08:11:50.879073  6167 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240503 08:11:50.879201  6167 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.029s	user 0.001s	sys 0.000s
I20240503 08:11:50.879326  6167 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240503 08:11:50.882781  6188 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:50.901963  6167 ts_tablet_manager.cc:590] Registered 12 tablets
I20240503 08:11:50.902096  6167 ts_tablet_manager.cc:569] Time spent register tablets: real 0.023s	user 0.016s	sys 0.003s
I20240503 08:11:50.903447  6188 log.cc:824] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:50.951274  6188 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:50.951895  6188 tablet_bootstrap.cc:492] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:50.953186  6188 ts_tablet_manager.cc:1367] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.071s	user 0.059s	sys 0.008s
W20240503 08:11:50.956379  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:11:50.965626  6188 raft_consensus.cc:348] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:50.967331  6188 raft_consensus.cc:729] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:50.968225  6188 consensus_queue.cc:260] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:50.970013  6188 ts_tablet_manager.cc:1398] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.017s	user 0.011s	sys 0.004s
I20240503 08:11:50.970563  6188 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:50.971231  6167 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:45423
I20240503 08:11:50.971299  6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:45423 every 8 connection(s)
I20240503 08:11:50.973001  6167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:50.979627  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 6167
I20240503 08:11:50.990384  6298 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42891
I20240503 08:11:50.990775  6298 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:50.991930  6298 heartbeater.cc:511] Master 127.2.49.190:42891 requested a full tablet report, sending...
I20240503 08:11:50.996351  2286 ts_manager.cc:194] Re-registered known tserver with Master: cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423)
I20240503 08:11:51.002197  2286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:40431
I20240503 08:11:51.018925  6188 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.019451  6188 tablet_bootstrap.cc:492] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.020514  6188 ts_tablet_manager.cc:1367] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.050s	user 0.031s	sys 0.016s
I20240503 08:11:51.021517  6188 raft_consensus.cc:348] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.021709  6188 raft_consensus.cc:729] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.021993  6188 consensus_queue.cc:260] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.022511  6188 ts_tablet_manager.cc:1398] T 4d06805616834a3ca83ed6bb5d6ad320 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.003s
I20240503 08:11:51.022598  6298 heartbeater.cc:503] Master 127.2.49.190:42891 was elected leader, sending a full tablet report...
I20240503 08:11:51.022841  6188 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.064675  6251 raft_consensus.cc:3037] T 5268a78550334d9b828888bc358ea695 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:51.071939  6188 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.072602  6188 tablet_bootstrap.cc:492] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.074131  6188 ts_tablet_manager.cc:1367] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.051s	user 0.042s	sys 0.005s
I20240503 08:11:51.074999  6188 raft_consensus.cc:348] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:51.075253  6188 raft_consensus.cc:729] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.075815  6188 consensus_queue.cc:260] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:51.076267  6188 ts_tablet_manager.cc:1398] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:11:51.076480  6188 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.128006  6188 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.128504  6188 tablet_bootstrap.cc:492] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.129633  6188 ts_tablet_manager.cc:1367] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.053s	user 0.034s	sys 0.016s
I20240503 08:11:51.130614  6188 raft_consensus.cc:348] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:51.130846  6188 raft_consensus.cc:729] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.131142  6188 consensus_queue.cc:260] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } }
I20240503 08:11:51.131618  6188 ts_tablet_manager.cc:1398] T 5da6fe49421748a598837fae0d0712e7 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:11:51.131997  6188 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.172123  6188 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.172598  6188 tablet_bootstrap.cc:492] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.173604  6188 ts_tablet_manager.cc:1367] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.042s	user 0.030s	sys 0.007s
I20240503 08:11:51.174417  6188 raft_consensus.cc:348] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.174610  6188 raft_consensus.cc:729] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.174845  6188 consensus_queue.cc:260] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.175341  6188 ts_tablet_manager.cc:1398] T 45e97b1190e742f1817c55b578728657 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:11:51.175702  6188 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.217808  6188 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.218412  6188 tablet_bootstrap.cc:492] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.219529  6188 ts_tablet_manager.cc:1367] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.044s	user 0.034s	sys 0.008s
I20240503 08:11:51.220302  6188 raft_consensus.cc:348] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:51.220484  6188 raft_consensus.cc:729] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.220767  6188 consensus_queue.cc:260] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:51.221202  6188 ts_tablet_manager.cc:1398] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:11:51.221580  6188 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.262058  6188 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.262533  6188 tablet_bootstrap.cc:492] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.263495  6188 ts_tablet_manager.cc:1367] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.042s	user 0.037s	sys 0.004s
I20240503 08:11:51.264171  6188 raft_consensus.cc:348] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:51.264393  6188 raft_consensus.cc:729] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.264631  6188 consensus_queue.cc:260] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } attrs { replace: true } } peers { permanent_uuid: "e4a4f8044a424243aa9ab82e0896099d" member_type: NON_VOTER last_known_addr { host: "127.2.49.135" port: 34325 } attrs { promote: true } }
I20240503 08:11:51.265192  6188 ts_tablet_manager.cc:1398] T 390e508ba635499da7e2d5b5ee56e718 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:11:51.265518  6188 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.285037  6251 raft_consensus.cc:3037] T 1b9906f233cd4c848eb58f2330ca70b3 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:51.306020  6188 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.306646  6188 tablet_bootstrap.cc:492] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.307718  6188 ts_tablet_manager.cc:1367] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.042s	user 0.029s	sys 0.011s
I20240503 08:11:51.308374  6188 raft_consensus.cc:348] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:51.308552  6188 raft_consensus.cc:729] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.308773  6188 consensus_queue.cc:260] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:51.309291  6188 ts_tablet_manager.cc:1398] T e9aa470143b74329b6f63d3ff1b1ec86 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:11:51.309604  6188 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.354503  6188 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.354976  6188 tablet_bootstrap.cc:492] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.356097  6188 ts_tablet_manager.cc:1367] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.047s	user 0.033s	sys 0.011s
I20240503 08:11:51.356817  6188 raft_consensus.cc:348] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:51.357023  6188 raft_consensus.cc:729] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.357280  6188 consensus_queue.cc:260] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } }
I20240503 08:11:51.357769  6188 ts_tablet_manager.cc:1398] T a587ae362dc34cde86d1053430ef8e44 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:11:51.358147  6188 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
W20240503 08:11:51.383344  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 41: this message will repeat every 5th retry.
I20240503 08:11:51.396008  6251 raft_consensus.cc:3037] T cd324aef02344416b643a2421ca46918 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:51.416957  6188 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.417495  6188 tablet_bootstrap.cc:492] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.418821  6188 ts_tablet_manager.cc:1367] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.061s	user 0.049s	sys 0.011s
I20240503 08:11:51.419680  6188 raft_consensus.cc:348] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:51.419929  6188 raft_consensus.cc:729] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.420225  6188 consensus_queue.cc:260] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } }
I20240503 08:11:51.421129  6188 ts_tablet_manager.cc:1398] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 08:11:51.421490  6188 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.469163  6188 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.469609  6188 tablet_bootstrap.cc:492] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.470619  6188 ts_tablet_manager.cc:1367] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.049s	user 0.033s	sys 0.015s
I20240503 08:11:51.471295  6188 raft_consensus.cc:348] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.471487  6188 raft_consensus.cc:729] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.471727  6188 consensus_queue.cc:260] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "f1443bb164fb4a79a008d9fcf3575b81" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 38461 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.472261  6188 ts_tablet_manager.cc:1398] T bbc60d8fc83f4cfab23bffcbd1eb72db P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:11:51.472551  6188 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap starting.
I20240503 08:11:51.531137  6188 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.531795  6188 tablet_bootstrap.cc:492] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Bootstrap complete.
I20240503 08:11:51.533329  6188 ts_tablet_manager.cc:1367] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Time spent bootstrapping tablet: real 0.061s	user 0.046s	sys 0.012s
I20240503 08:11:51.534406  6188 raft_consensus.cc:348] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.534664  6188 raft_consensus.cc:729] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc209b43de14ccbbba8e8770cb02d47, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.534998  6188 consensus_queue.cc:260] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15068fdaeae4d89b40ed67c3f37a30e" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 40299 } } peers { permanent_uuid: "032a50802ce84632bfb57b6a1a840606" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34755 } } peers { permanent_uuid: "cfc209b43de14ccbbba8e8770cb02d47" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 45423 } }
I20240503 08:11:51.535822  6188 ts_tablet_manager.cc:1398] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:11:51.644434  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:51.649483  6231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:51.651387  2800 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:51.667754  2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:51.678902  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:51.679842  4164 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:51.715507  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:51.735352  6251 raft_consensus.cc:3037] T 22756ba18d2f49999f02ac878b8bf03d P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:51.770838  2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:51.945291  2800 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:51.946559  6251 raft_consensus.cc:3037] T a62c007e061d4a96b4a3c8633bd29ee5 P cfc209b43de14ccbbba8e8770cb02d47 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:51.980481  3902 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:51.990031  6231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:51.993686  2539 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:52.009493  4033 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:52.012985  4164 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:52.024555  4294 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:52.040491  2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:52.263213  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 6167
I20240503 08:11:52.271044  6282 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:52.383020  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 6167
I20240503 08:11:52.405886  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2479
W20240503 08:11:52.416406  2626 consensus_peers.cc:485] T 45e97b1190e742f1817c55b578728657 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:52.417483  2600 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:52.519968  2626 consensus_peers.cc:485] T bbc60d8fc83f4cfab23bffcbd1eb72db P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:52.528376  2756 consensus_peers.cc:485] T a62c007e061d4a96b4a3c8633bd29ee5 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:52.576623  2626 consensus_peers.cc:485] T 5da6fe49421748a598837fae0d0712e7 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:52.584136  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2479
W20240503 08:11:52.623482  2626 consensus_peers.cc:485] T a587ae362dc34cde86d1053430ef8e44 P f1443bb164fb4a79a008d9fcf3575b81 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:52.624516  4117 connection.cc:535] client connection to 127.2.49.130:40299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:52.624742  4117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:52.624982  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2609
W20240503 08:11:52.624980  2757 connection.cc:535] server connection from 127.2.49.130:44251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:52.627169  4117 consensus_peers.cc:485] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d -> Peer a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299): Couldn't send request to peer a15068fdaeae4d89b40ed67c3f37a30e. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:40299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:52.631361  2730 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:52.687525  2756 consensus_peers.cc:485] T 1b9906f233cd4c848eb58f2330ca70b3 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:52.776260  2754 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer a15068fdaeae4d89b40ed67c3f37a30e (127.2.49.130:40299): Couldn't send request to peer a15068fdaeae4d89b40ed67c3f37a30e. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:40299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:52.800756  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2609
W20240503 08:11:52.830839  2754 connection.cc:535] server connection from 127.2.49.131:40449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:52.830993  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2739
I20240503 08:11:52.837433  2860 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:52.996762  2756 consensus_peers.cc:485] T cd324aef02344416b643a2421ca46918 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:52.997767  2756 consensus_peers.cc:485] T 22756ba18d2f49999f02ac878b8bf03d P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:52.999243  2756 consensus_peers.cc:485] T 4d06805616834a3ca83ed6bb5d6ad320 P 032a50802ce84632bfb57b6a1a840606 -> Peer cfc209b43de14ccbbba8e8770cb02d47 (127.2.49.129:45423): Couldn't send request to peer cfc209b43de14ccbbba8e8770cb02d47. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:53.000662  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2739
I20240503 08:11:53.030102  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 3840
I20240503 08:11:53.036351  3961 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:53.169245  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 3840
I20240503 08:11:53.192912  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 3971
I20240503 08:11:53.199162  4093 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:53.292515  4118 consensus_peers.cc:485] T 390e508ba635499da7e2d5b5ee56e718 P e4a4f8044a424243aa9ab82e0896099d -> Peer 1cc62a1159dd4742b2a609e4756443f2 (127.2.49.133:45149): Couldn't send request to peer 1cc62a1159dd4742b2a609e4756443f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:45149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:53.321520  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 3971
I20240503 08:11:53.344043  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4101
I20240503 08:11:53.350323  4223 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:53.484597  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4101
I20240503 08:11:53.511349  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4232
I20240503 08:11:53.518023  4353 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:53.635035  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4232
I20240503 08:11:53.652432  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2258
I20240503 08:11:53.653510  2318 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:53.814946  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2258
2024-05-03T08:11:53Z chronyd exiting
[       OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (46874 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (46874 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240503 08:11:53.884311  2246 test_util.cc:268] Using random seed: 1170629231
I20240503 08:11:53.886134  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:53.886284  2246 ts_itest-base.cc:113] --------------
I20240503 08:11:53.886343  2246 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:53.886413  2246 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:53.886462  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:53Z Disabled control of system clock
I20240503 08:11:53.911231  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:45493
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:45493
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 08:11:54.242848  6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:54.243180  6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:54.243302  6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:54.250790  6455 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:54.250921  6455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:54.250993  6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:54.251078  6455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:54.251140  6455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:54.260246  6455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:45493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:54.261597  6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:54.263335  6455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:54.274519  6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.274987  6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.275964  6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:54.276908  6455 server_base.cc:1034] running on GCE node
I20240503 08:11:54.277755  6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:54.279323  6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:54.280521  6455 hybrid_clock.cc:648] HybridClock initialized: now 1714723914280483 us; error 39 us; skew 500 ppm
I20240503 08:11:54.280885  6455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:54.283231  6455 webserver.cc:460] Webserver started at http://127.2.49.190:39721/ using document root <none> and password file <none>
I20240503 08:11:54.283972  6455 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:54.284090  6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:54.284415  6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:54.287184  6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d260cea03c6246ba9b469df97854748e"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dz87"
I20240503 08:11:54.287987  6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d260cea03c6246ba9b469df97854748e"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dz87"
I20240503 08:11:54.293453  6455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240503 08:11:54.297070  6469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:54.298538  6455 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:54.298722  6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d260cea03c6246ba9b469df97854748e"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dz87"
I20240503 08:11:54.298918  6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:54.317905  6455 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:54.318992  6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:54.319331  6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:54.338353  6455 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:45493
I20240503 08:11:54.338376  6520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:45493 every 8 connection(s)
I20240503 08:11:54.339779  6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:54.343709  6521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.349802  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 6455
I20240503 08:11:54.350081  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:54.354616  6521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e: Bootstrap starting.
I20240503 08:11:54.358062  6521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:54.359200  6521 log.cc:824] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:54.362387  6521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e: No bootstrap required, opened a new log
I20240503 08:11:54.369681  6521 raft_consensus.cc:348] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d260cea03c6246ba9b469df97854748e" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 45493 } }
I20240503 08:11:54.370211  6521 raft_consensus.cc:374] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:54.370359  6521 raft_consensus.cc:729] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d260cea03c6246ba9b469df97854748e, State: Initialized, Role: FOLLOWER
I20240503 08:11:54.371052  6521 consensus_queue.cc:260] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d260cea03c6246ba9b469df97854748e" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 45493 } }
I20240503 08:11:54.371277  6521 raft_consensus.cc:388] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:54.371402  6521 raft_consensus.cc:482] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:54.371548  6521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:54.373944  6521 raft_consensus.cc:504] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d260cea03c6246ba9b469df97854748e" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 45493 } }
I20240503 08:11:54.374509  6521 leader_election.cc:302] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d260cea03c6246ba9b469df97854748e; no voters: 
I20240503 08:11:54.375154  6521 leader_election.cc:288] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:11:54.375315  6526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:54.375955  6526 raft_consensus.cc:686] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [term 1 LEADER]: Becoming Leader. State: Replica: d260cea03c6246ba9b469df97854748e, State: Running, Role: LEADER
I20240503 08:11:54.376572  6526 consensus_queue.cc:237] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d260cea03c6246ba9b469df97854748e" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 45493 } }
I20240503 08:11:54.377403  6521 sys_catalog.cc:564] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:54.379675  6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d260cea03c6246ba9b469df97854748e. Latest consensus state: current_term: 1 leader_uuid: "d260cea03c6246ba9b469df97854748e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d260cea03c6246ba9b469df97854748e" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 45493 } } }
I20240503 08:11:54.379951  6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:54.381573  6532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:54.381543  6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d260cea03c6246ba9b469df97854748e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d260cea03c6246ba9b469df97854748e" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 45493 } } }
I20240503 08:11:54.381860  6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:54.387786  6532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:54.396625  6532 catalog_manager.cc:1347] Generated new cluster ID: 654f766dc1444797a7db63c04d621ea7
I20240503 08:11:54.396760  6532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:54.416618  6532 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:54.418412  6532 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:54.439697  6532 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d260cea03c6246ba9b469df97854748e: Generated new TSK 0
I20240503 08:11:54.440644  6532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:54.455358  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--builtin_ntp_servers=127.2.49.148:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:11:54.844599  6545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:54.844913  6545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:54.845034  6545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:54.853765  6545 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:54.853943  6545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:54.854285  6545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:11:54.863837  6545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:54.865211  6545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:54.867012  6545 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:54.879930  6553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.880441  6551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.880930  6550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:54.881932  6545 server_base.cc:1034] running on GCE node
I20240503 08:11:54.882652  6545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:54.883692  6545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:54.884919  6545 hybrid_clock.cc:648] HybridClock initialized: now 1714723914884864 us; error 46 us; skew 500 ppm
I20240503 08:11:54.885237  6545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:54.887719  6545 webserver.cc:460] Webserver started at http://127.2.49.129:36271/ using document root <none> and password file <none>
I20240503 08:11:54.888463  6545 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:54.888563  6545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:54.888893  6545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:54.891597  6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "036d6f34e25c4764a09ca129f00d9630"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dz87"
I20240503 08:11:54.892380  6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "036d6f34e25c4764a09ca129f00d9630"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dz87"
I20240503 08:11:54.897589  6545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 08:11:54.901201  6559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:54.902721  6545 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:54.902889  6545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "036d6f34e25c4764a09ca129f00d9630"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dz87"
I20240503 08:11:54.903118  6545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:54.925179  6545 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:54.926116  6545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:54.926436  6545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:54.927461  6545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:54.928978  6545 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:54.929067  6545 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:54.929149  6545 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:54.929208  6545 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:54.961230  6545 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:44827
I20240503 08:11:54.961313  6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:44827 every 8 connection(s)
I20240503 08:11:54.962647  6545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:54.967617  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 6545
I20240503 08:11:54.967816  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:54.971282  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--builtin_ntp_servers=127.2.49.148:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:54.976495  6672 heartbeater.cc:347] Connected to a master server at 127.2.49.190:45493
I20240503 08:11:54.976795  6672 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:54.977543  6672 heartbeater.cc:511] Master 127.2.49.190:45493 requested a full tablet report, sending...
I20240503 08:11:54.979874  6486 ts_manager.cc:194] Registered new tserver with Master: 036d6f34e25c4764a09ca129f00d9630 (127.2.49.129:44827)
I20240503 08:11:54.982221  6486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:51079
W20240503 08:11:55.313405  6676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:55.313715  6676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:55.313803  6676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:55.321442  6676 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:55.321574  6676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:55.321707  6676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:11:55.330969  6676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:55.332310  6676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:55.334079  6676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:55.348311  6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.349545  6682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.350272  6684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:55.351158  6676 server_base.cc:1034] running on GCE node
I20240503 08:11:55.351917  6676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:55.353127  6676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:55.354370  6676 hybrid_clock.cc:648] HybridClock initialized: now 1714723915354300 us; error 65 us; skew 500 ppm
I20240503 08:11:55.354764  6676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:55.357757  6676 webserver.cc:460] Webserver started at http://127.2.49.130:34791/ using document root <none> and password file <none>
I20240503 08:11:55.358574  6676 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:55.358680  6676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:55.358963  6676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:55.361620  6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e75e5671266f4bb5a46be0bf8b42aa52"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dz87"
I20240503 08:11:55.362471  6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e75e5671266f4bb5a46be0bf8b42aa52"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dz87"
I20240503 08:11:55.367890  6676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:11:55.371626  6690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:55.373072  6676 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:11:55.373222  6676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e75e5671266f4bb5a46be0bf8b42aa52"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dz87"
I20240503 08:11:55.373409  6676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:55.398808  6676 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:55.399641  6676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:55.399897  6676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:55.400887  6676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:55.402395  6676 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:55.402483  6676 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:55.402567  6676 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:55.402621  6676 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:55.435696  6676 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:43505
I20240503 08:11:55.435753  6802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:43505 every 8 connection(s)
I20240503 08:11:55.437093  6676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:55.442603  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 6676
I20240503 08:11:55.442829  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:55.446602  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--builtin_ntp_servers=127.2.49.148:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:55.450984  6803 heartbeater.cc:347] Connected to a master server at 127.2.49.190:45493
I20240503 08:11:55.451324  6803 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:55.452095  6803 heartbeater.cc:511] Master 127.2.49.190:45493 requested a full tablet report, sending...
I20240503 08:11:55.453661  6486 ts_manager.cc:194] Registered new tserver with Master: e75e5671266f4bb5a46be0bf8b42aa52 (127.2.49.130:43505)
I20240503 08:11:55.454875  6486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:46205
W20240503 08:11:55.789075  6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:55.789381  6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:55.789469  6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:55.797081  6806 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:55.797214  6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:55.797331  6806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:11:55.806653  6806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:55.807992  6806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:55.809773  6806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:55.822595  6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.823055  6814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.823652  6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:55.824484  6806 server_base.cc:1034] running on GCE node
I20240503 08:11:55.825170  6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:55.826224  6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:55.827454  6806 hybrid_clock.cc:648] HybridClock initialized: now 1714723915827410 us; error 44 us; skew 500 ppm
I20240503 08:11:55.827804  6806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:55.830484  6806 webserver.cc:460] Webserver started at http://127.2.49.131:46113/ using document root <none> and password file <none>
I20240503 08:11:55.831303  6806 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:55.831411  6806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:55.831720  6806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:55.834703  6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e52540f47a7b406a885e75c3697fcc3d"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dz87"
I20240503 08:11:55.835520  6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e52540f47a7b406a885e75c3697fcc3d"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dz87"
I20240503 08:11:55.841279  6806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:11:55.845198  6820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:55.846845  6806 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:11:55.847004  6806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e52540f47a7b406a885e75c3697fcc3d"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dz87"
I20240503 08:11:55.847193  6806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:55.865672  6806 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:55.866693  6806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:55.866948  6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:55.867915  6806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:55.869547  6806 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:55.869647  6806 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:55.869761  6806 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:55.869817  6806 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:55.901947  6806 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:35563
I20240503 08:11:55.902002  6932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:35563 every 8 connection(s)
I20240503 08:11:55.903537  6806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:55.907696  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 6806
I20240503 08:11:55.907879  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:55.911501  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--builtin_ntp_servers=127.2.49.148:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:55.918454  6933 heartbeater.cc:347] Connected to a master server at 127.2.49.190:45493
I20240503 08:11:55.918807  6933 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:55.919585  6933 heartbeater.cc:511] Master 127.2.49.190:45493 requested a full tablet report, sending...
I20240503 08:11:55.921101  6486 ts_manager.cc:194] Registered new tserver with Master: e52540f47a7b406a885e75c3697fcc3d (127.2.49.131:35563)
I20240503 08:11:55.921905  6486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:36045
I20240503 08:11:55.985688  6672 heartbeater.cc:503] Master 127.2.49.190:45493 was elected leader, sending a full tablet report...
W20240503 08:11:56.254720  6936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:56.255060  6936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:56.255199  6936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:56.262825  6936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:56.262990  6936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:56.263157  6936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:11:56.272469  6936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:56.273864  6936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:56.275648  6936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:56.288586  6942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.289137  6941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.289806  6944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:56.290956  6936 server_base.cc:1034] running on GCE node
I20240503 08:11:56.291615  6936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:56.292577  6936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:56.293802  6936 hybrid_clock.cc:648] HybridClock initialized: now 1714723916293744 us; error 48 us; skew 500 ppm
I20240503 08:11:56.294196  6936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:56.296664  6936 webserver.cc:460] Webserver started at http://127.2.49.132:43109/ using document root <none> and password file <none>
I20240503 08:11:56.297432  6936 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:56.297536  6936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:56.297853  6936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:56.300601  6936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "aa8574ea99414b6db70a89d567c9e4a7"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dz87"
I20240503 08:11:56.301420  6936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "aa8574ea99414b6db70a89d567c9e4a7"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dz87"
I20240503 08:11:56.307127  6936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240503 08:11:56.311038  6950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:56.312543  6936 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:11:56.312729  6936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "aa8574ea99414b6db70a89d567c9e4a7"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dz87"
I20240503 08:11:56.312958  6936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:56.331373  6936 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:56.332263  6936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:56.332551  6936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:56.333539  6936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:56.335139  6936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:56.335243  6936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:56.335359  6936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:56.335430  6936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:56.368820  6936 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:35799
I20240503 08:11:56.368875  7062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:35799 every 8 connection(s)
I20240503 08:11:56.370283  6936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:56.372277  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 6936
I20240503 08:11:56.372511  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:56.375762  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:0
--local_ip_for_outbound_sockets=127.2.49.133
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--builtin_ntp_servers=127.2.49.148:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:11:56.385653  7063 heartbeater.cc:347] Connected to a master server at 127.2.49.190:45493
I20240503 08:11:56.386008  7063 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:56.386875  7063 heartbeater.cc:511] Master 127.2.49.190:45493 requested a full tablet report, sending...
I20240503 08:11:56.388378  6486 ts_manager.cc:194] Registered new tserver with Master: aa8574ea99414b6db70a89d567c9e4a7 (127.2.49.132:35799)
I20240503 08:11:56.389240  6486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:49761
I20240503 08:11:56.457820  6803 heartbeater.cc:503] Master 127.2.49.190:45493 was elected leader, sending a full tablet report...
W20240503 08:11:56.719609  7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:56.719923  7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:56.720012  7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:56.727691  7066 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:11:56.727835  7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:56.727952  7066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:11:56.737327  7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:45493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:56.738683  7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:56.740424  7066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:56.752976  7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.753376  7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.754350  7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:56.754974  7066 server_base.cc:1034] running on GCE node
I20240503 08:11:56.755599  7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:56.756561  7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:56.757781  7066 hybrid_clock.cc:648] HybridClock initialized: now 1714723916757738 us; error 30 us; skew 500 ppm
I20240503 08:11:56.758277  7066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:56.760713  7066 webserver.cc:460] Webserver started at http://127.2.49.133:33653/ using document root <none> and password file <none>
I20240503 08:11:56.761435  7066 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:56.761545  7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:56.761830  7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:56.764509  7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "830a9c17dc44494e8ebf622b67eda14d"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dz87"
I20240503 08:11:56.765292  7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "830a9c17dc44494e8ebf622b67eda14d"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dz87"
I20240503 08:11:56.770752  7066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 08:11:56.774291  7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:56.775878  7066 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:11:56.776037  7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "830a9c17dc44494e8ebf622b67eda14d"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dz87"
I20240503 08:11:56.776254  7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:56.801008  7066 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:11:56.801898  7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:56.802201  7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:56.803259  7066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:56.804868  7066 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:56.804967  7066 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:56.805045  7066 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:56.805107  7066 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:11:56.838682  7066 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:36919
I20240503 08:11:56.838747  7192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:36919 every 8 connection(s)
I20240503 08:11:56.840023  7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:56.847493  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 7066
I20240503 08:11:56.847707  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:56.853870  7193 heartbeater.cc:347] Connected to a master server at 127.2.49.190:45493
I20240503 08:11:56.854216  7193 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:56.855032  7193 heartbeater.cc:511] Master 127.2.49.190:45493 requested a full tablet report, sending...
I20240503 08:11:56.856385  6486 ts_manager.cc:194] Registered new tserver with Master: 830a9c17dc44494e8ebf622b67eda14d (127.2.49.133:36919)
I20240503 08:11:56.857180  6486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:52771
I20240503 08:11:56.864049  2246 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:56.881677  6486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40826:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 08:11:56.918038  6738 tablet_service.cc:1467] Processing CreateTablet for tablet c9d22fea7f1d4171a2d170af0ec7ac35 (DEFAULT_TABLE table=TestTable [id=d2b033fb145441ae92160dba18239106]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:56.919852  6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d22fea7f1d4171a2d170af0ec7ac35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:56.924527  6933 heartbeater.cc:503] Master 127.2.49.190:45493 was elected leader, sending a full tablet report...
I20240503 08:11:56.924407  6998 tablet_service.cc:1467] Processing CreateTablet for tablet c9d22fea7f1d4171a2d170af0ec7ac35 (DEFAULT_TABLE table=TestTable [id=d2b033fb145441ae92160dba18239106]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:56.924929  6868 tablet_service.cc:1467] Processing CreateTablet for tablet c9d22fea7f1d4171a2d170af0ec7ac35 (DEFAULT_TABLE table=TestTable [id=d2b033fb145441ae92160dba18239106]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:56.926379  6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d22fea7f1d4171a2d170af0ec7ac35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:56.926427  6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d22fea7f1d4171a2d170af0ec7ac35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:56.934684  7211 tablet_bootstrap.cc:492] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: Bootstrap starting.
I20240503 08:11:56.937961  7212 tablet_bootstrap.cc:492] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: Bootstrap starting.
I20240503 08:11:56.939843  7211 tablet_bootstrap.cc:654] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:56.939834  7213 tablet_bootstrap.cc:492] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: Bootstrap starting.
I20240503 08:11:56.941329  7211 log.cc:824] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:56.941391  7212 tablet_bootstrap.cc:654] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:56.942595  7212 log.cc:824] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:56.943624  7213 tablet_bootstrap.cc:654] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:56.944923  7213 log.cc:824] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:56.945073  7211 tablet_bootstrap.cc:492] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: No bootstrap required, opened a new log
I20240503 08:11:56.945410  7211 ts_tablet_manager.cc:1367] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.002s
I20240503 08:11:56.948129  7213 tablet_bootstrap.cc:492] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: No bootstrap required, opened a new log
I20240503 08:11:56.948544  7213 ts_tablet_manager.cc:1367] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240503 08:11:56.953558  7212 tablet_bootstrap.cc:492] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: No bootstrap required, opened a new log
I20240503 08:11:56.953842  7212 ts_tablet_manager.cc:1367] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.008s
I20240503 08:11:56.955418  7211 raft_consensus.cc:348] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:56.955902  7211 raft_consensus.cc:374] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:56.956046  7211 raft_consensus.cc:729] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75e5671266f4bb5a46be0bf8b42aa52, State: Initialized, Role: FOLLOWER
I20240503 08:11:56.956792  7211 consensus_queue.cc:260] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:56.957368  7213 raft_consensus.cc:348] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:56.957883  7213 raft_consensus.cc:374] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:56.958029  7213 raft_consensus.cc:729] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52540f47a7b406a885e75c3697fcc3d, State: Initialized, Role: FOLLOWER
I20240503 08:11:56.958724  7211 ts_tablet_manager.cc:1398] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.002s
I20240503 08:11:56.958938  7213 consensus_queue.cc:260] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:56.960759  7213 ts_tablet_manager.cc:1398] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20240503 08:11:56.961259  7212 raft_consensus.cc:348] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:56.961701  7212 raft_consensus.cc:374] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:56.961809  7212 raft_consensus.cc:729] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8574ea99414b6db70a89d567c9e4a7, State: Initialized, Role: FOLLOWER
I20240503 08:11:56.962517  7212 consensus_queue.cc:260] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:56.966616  7063 heartbeater.cc:503] Master 127.2.49.190:45493 was elected leader, sending a full tablet report...
I20240503 08:11:56.967209  7212 ts_tablet_manager.cc:1398] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.008s
I20240503 08:11:57.051229  7218 raft_consensus.cc:482] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:57.051486  7218 raft_consensus.cc:504] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:57.053242  7218 leader_election.cc:288] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e75e5671266f4bb5a46be0bf8b42aa52 (127.2.49.130:43505), aa8574ea99414b6db70a89d567c9e4a7 (127.2.49.132:35799)
I20240503 08:11:57.062567  6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d22fea7f1d4171a2d170af0ec7ac35" candidate_uuid: "e52540f47a7b406a885e75c3697fcc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" is_pre_election: true
I20240503 08:11:57.062631  7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d22fea7f1d4171a2d170af0ec7ac35" candidate_uuid: "e52540f47a7b406a885e75c3697fcc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8574ea99414b6db70a89d567c9e4a7" is_pre_election: true
I20240503 08:11:57.063092  6758 raft_consensus.cc:2454] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e52540f47a7b406a885e75c3697fcc3d in term 0.
I20240503 08:11:57.063084  7018 raft_consensus.cc:2454] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e52540f47a7b406a885e75c3697fcc3d in term 0.
I20240503 08:11:57.063828  6821 leader_election.cc:302] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa8574ea99414b6db70a89d567c9e4a7, e52540f47a7b406a885e75c3697fcc3d; no voters: 
I20240503 08:11:57.064330  7218 raft_consensus.cc:2781] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:57.064570  7218 raft_consensus.cc:482] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:57.064687  7218 raft_consensus.cc:3037] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:57.066900  7218 raft_consensus.cc:504] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:57.067726  7218 leader_election.cc:288] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [CANDIDATE]: Term 1 election: Requested vote from peers e75e5671266f4bb5a46be0bf8b42aa52 (127.2.49.130:43505), aa8574ea99414b6db70a89d567c9e4a7 (127.2.49.132:35799)
I20240503 08:11:57.068289  7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d22fea7f1d4171a2d170af0ec7ac35" candidate_uuid: "e52540f47a7b406a885e75c3697fcc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8574ea99414b6db70a89d567c9e4a7"
I20240503 08:11:57.068359  6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d22fea7f1d4171a2d170af0ec7ac35" candidate_uuid: "e52540f47a7b406a885e75c3697fcc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75e5671266f4bb5a46be0bf8b42aa52"
I20240503 08:11:57.068499  7018 raft_consensus.cc:3037] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:57.068578  6758 raft_consensus.cc:3037] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:57.071187  7018 raft_consensus.cc:2454] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e52540f47a7b406a885e75c3697fcc3d in term 1.
I20240503 08:11:57.071213  6758 raft_consensus.cc:2454] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e52540f47a7b406a885e75c3697fcc3d in term 1.
I20240503 08:11:57.071671  6821 leader_election.cc:302] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa8574ea99414b6db70a89d567c9e4a7, e52540f47a7b406a885e75c3697fcc3d; no voters: 
I20240503 08:11:57.072033  7218 raft_consensus.cc:2781] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:57.072492  7218 raft_consensus.cc:686] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [term 1 LEADER]: Becoming Leader. State: Replica: e52540f47a7b406a885e75c3697fcc3d, State: Running, Role: LEADER
I20240503 08:11:57.073005  7218 consensus_queue.cc:237] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } }
I20240503 08:11:57.077786  6484 catalog_manager.cc:5484] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d reported cstate change: term changed from 0 to 1, leader changed from <none> to e52540f47a7b406a885e75c3697fcc3d (127.2.49.131). New cstate: current_term: 1 leader_uuid: "e52540f47a7b406a885e75c3697fcc3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 } } peers { permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 } } peers { permanent_uuid: "e52540f47a7b406a885e75c3697fcc3d" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 35563 } } }
I20240503 08:11:57.116613  2246 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:57.119707  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 036d6f34e25c4764a09ca129f00d9630 to finish bootstrapping
W20240503 08:11:57.122421  7064 tablet.cc:2369] T c9d22fea7f1d4171a2d170af0ec7ac35 P aa8574ea99414b6db70a89d567c9e4a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:57.130343  2246 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e75e5671266f4bb5a46be0bf8b42aa52 to finish bootstrapping
I20240503 08:11:57.138829  2246 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e52540f47a7b406a885e75c3697fcc3d to finish bootstrapping
I20240503 08:11:57.146937  2246 ts_itest-base.cc:233] Waiting for 1 tablets on tserver aa8574ea99414b6db70a89d567c9e4a7 to finish bootstrapping
I20240503 08:11:57.155408  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 830a9c17dc44494e8ebf622b67eda14d to finish bootstrapping
W20240503 08:11:57.156221  6934 tablet.cc:2369] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:57.190050  6804 tablet.cc:2369] T c9d22fea7f1d4171a2d170af0ec7ac35 P e75e5671266f4bb5a46be0bf8b42aa52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:57.513496  7218 consensus_queue.cc:1035] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8574ea99414b6db70a89d567c9e4a7" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:57.522064  7218 consensus_queue.cc:1035] T c9d22fea7f1d4171a2d170af0ec7ac35 P e52540f47a7b406a885e75c3697fcc3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75e5671266f4bb5a46be0bf8b42aa52" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:57.661792  6607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.670306  6998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.671269  7128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.678586  6868 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:57.682929  6738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.860646  7193 heartbeater.cc:503] Master 127.2.49.190:45493 was elected leader, sending a full tablet report...
I20240503 08:11:57.868847  6607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.871755  6998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.872198  7128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:57.892426  6868 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:57.909574  6738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.552552  7128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.556588  6607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.562500  6998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.563592  6738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.579037  6868 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:58.736245  6607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.743973  7128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.746330  6868 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:58.748005  6998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.766625  6738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.934927  6607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.946003  6998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.953574  7128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:58.967924  6868 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:58.973580  6738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:59.128074  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 6545
I20240503 08:11:59.129783  6666 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.221407  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 6545
I20240503 08:11:59.239660  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 6676
I20240503 08:11:59.240695  6797 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.338726  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 6676
I20240503 08:11:59.357841  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 6806
I20240503 08:11:59.358879  6927 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.461968  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 6806
I20240503 08:11:59.480579  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 6936
I20240503 08:11:59.481734  7058 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.577050  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 6936
W20240503 08:11:59.597405  6473 connection.cc:535] server connection from 127.2.49.132:49761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:59.597641  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 7066
I20240503 08:11:59.598778  7187 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.689878  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 7066
I20240503 08:11:59.707767  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 6455
I20240503 08:11:59.709013  6516 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:59.824198  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 6455
2024-05-03T08:11:59Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5985 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5985 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/2
I20240503 08:11:59.869582  2246 test_util.cc:268] Using random seed: 1176614510
I20240503 08:11:59.872116  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:59.872216  2246 ts_itest-base.cc:113] --------------
I20240503 08:11:59.872285  2246 ts_itest-base.cc:114] 7 tablet servers
I20240503 08:11:59.872341  2246 ts_itest-base.cc:115] 2 replicas per TS
I20240503 08:11:59.872411  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:59Z Disabled control of system clock
I20240503 08:11:59.899747  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:41099
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:41099
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 08:12:00.247934  7421 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:12:00.248347  7421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:00.248461  7421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:00.248579  7421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:00.256314  7421 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:00.256475  7421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:12:00.256592  7421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:00.256662  7421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:12:00.256762  7421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:12:00.266322  7421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:41099
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:00.267735  7421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:00.269524  7421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:00.281340  7429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:00.282295  7427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:00.282830  7426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:00.283706  7421 server_base.cc:1034] running on GCE node
I20240503 08:12:00.284555  7421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:00.286121  7421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:00.287395  7421 hybrid_clock.cc:648] HybridClock initialized: now 1714723920287335 us; error 51 us; skew 500 ppm
I20240503 08:12:00.287748  7421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:00.290375  7421 webserver.cc:460] Webserver started at http://127.2.49.190:38405/ using document root <none> and password file <none>
I20240503 08:12:00.291168  7421 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:00.291311  7421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:00.291618  7421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:00.294533  7421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "56ef465a043845eb8a012eb5e7ce8be4"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-dz87"
I20240503 08:12:00.295333  7421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "56ef465a043845eb8a012eb5e7ce8be4"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-dz87"
I20240503 08:12:00.301379  7421 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240503 08:12:00.305294  7435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:00.306855  7421 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:12:00.307066  7421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "56ef465a043845eb8a012eb5e7ce8be4"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-dz87"
I20240503 08:12:00.307276  7421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:00.319909  7421 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:00.320809  7421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:00.321122  7421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:00.339699  7421 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:41099
I20240503 08:12:00.339722  7486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:41099 every 8 connection(s)
I20240503 08:12:00.341212  7421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:12:00.344745  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 7421
I20240503 08:12:00.344980  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:12:00.345965  7487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:00.359596  7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4: Bootstrap starting.
I20240503 08:12:00.362828  7487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:00.363919  7487 log.cc:824] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:00.366740  7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4: No bootstrap required, opened a new log
I20240503 08:12:00.373293  7487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ef465a043845eb8a012eb5e7ce8be4" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 41099 } }
I20240503 08:12:00.373692  7487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:00.373798  7487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ef465a043845eb8a012eb5e7ce8be4, State: Initialized, Role: FOLLOWER
I20240503 08:12:00.374483  7487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ef465a043845eb8a012eb5e7ce8be4" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 41099 } }
I20240503 08:12:00.374727  7487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:00.374882  7487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:00.375123  7487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:00.377277  7487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ef465a043845eb8a012eb5e7ce8be4" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 41099 } }
I20240503 08:12:00.377822  7487 leader_election.cc:302] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56ef465a043845eb8a012eb5e7ce8be4; no voters: 
I20240503 08:12:00.378486  7487 leader_election.cc:288] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:12:00.378655  7492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:00.379319  7492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [term 1 LEADER]: Becoming Leader. State: Replica: 56ef465a043845eb8a012eb5e7ce8be4, State: Running, Role: LEADER
I20240503 08:12:00.379948  7492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ef465a043845eb8a012eb5e7ce8be4" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 41099 } }
I20240503 08:12:00.380575  7487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:00.382917  7494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ef465a043845eb8a012eb5e7ce8be4. Latest consensus state: current_term: 1 leader_uuid: "56ef465a043845eb8a012eb5e7ce8be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ef465a043845eb8a012eb5e7ce8be4" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 41099 } } }
I20240503 08:12:00.383213  7494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:00.384178  7497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:00.383963  7493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56ef465a043845eb8a012eb5e7ce8be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ef465a043845eb8a012eb5e7ce8be4" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 41099 } } }
I20240503 08:12:00.385324  7493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:00.390437  7497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:00.399753  7497 catalog_manager.cc:1347] Generated new cluster ID: ecbf686498b041ca85bcfd1fc245bf5b
I20240503 08:12:00.399907  7497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:00.424140  7497 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:00.425397  7497 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:00.435000  7497 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 56ef465a043845eb8a012eb5e7ce8be4: Generated new TSK 0
I20240503 08:12:00.435925  7497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:00.445042  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:12:00.797348  7511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:00.797677  7511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:00.797767  7511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:00.806017  7511 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:00.806207  7511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:00.806360  7511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:12:00.816073  7511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:00.817417  7511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:00.819401  7511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:00.832610  7516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:00.833330  7519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:00.833978  7517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:00.834925  7511 server_base.cc:1034] running on GCE node
I20240503 08:12:00.835579  7511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:00.836616  7511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:00.837827  7511 hybrid_clock.cc:648] HybridClock initialized: now 1714723920837786 us; error 45 us; skew 500 ppm
I20240503 08:12:00.838188  7511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:00.840634  7511 webserver.cc:460] Webserver started at http://127.2.49.129:37771/ using document root <none> and password file <none>
I20240503 08:12:00.841416  7511 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:00.841548  7511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:00.841867  7511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:00.844641  7511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "55fc889503854003801521bf97120179"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-dz87"
I20240503 08:12:00.845490  7511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "55fc889503854003801521bf97120179"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-dz87"
I20240503 08:12:00.851058  7511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.004s
I20240503 08:12:00.854820  7525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:00.856442  7511 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:12:00.856638  7511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "55fc889503854003801521bf97120179"
format_stamp: "Formatted at 2024-05-03 08:12:00 on dist-test-slave-dz87"
I20240503 08:12:00.856858  7511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:00.879484  7511 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:00.880466  7511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:00.880795  7511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:00.881947  7511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:00.883601  7511 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:00.883710  7511 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:00.883828  7511 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:00.883909  7511 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:00.917253  7511 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:39639
I20240503 08:12:00.917263  7637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:39639 every 8 connection(s)
I20240503 08:12:00.918749  7511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:12:00.927042  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 7511
I20240503 08:12:00.927258  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:12:00.931139  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:00.933967  7638 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:00.934391  7638 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:00.935407  7638 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:00.938107  7452 ts_manager.cc:194] Registered new tserver with Master: 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:00.940477  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:35887
W20240503 08:12:01.275401  7642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:01.275717  7642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:01.275810  7642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:01.283543  7642 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:01.283682  7642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:01.283807  7642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:12:01.293076  7642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:01.294459  7642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:01.296303  7642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:01.308964  7648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.309579  7647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.310346  7650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:01.310870  7642 server_base.cc:1034] running on GCE node
I20240503 08:12:01.311587  7642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:01.312723  7642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:01.313992  7642 hybrid_clock.cc:648] HybridClock initialized: now 1714723921313915 us; error 70 us; skew 500 ppm
I20240503 08:12:01.314416  7642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:01.316797  7642 webserver.cc:460] Webserver started at http://127.2.49.130:36629/ using document root <none> and password file <none>
I20240503 08:12:01.317600  7642 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:01.317728  7642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:01.318050  7642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:01.320868  7642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "359f2f323e79431eac07dc0a3f47a47b"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-dz87"
I20240503 08:12:01.321748  7642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "359f2f323e79431eac07dc0a3f47a47b"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-dz87"
I20240503 08:12:01.327397  7642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20240503 08:12:01.331279  7656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:01.332917  7642 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:12:01.333158  7642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "359f2f323e79431eac07dc0a3f47a47b"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-dz87"
I20240503 08:12:01.333392  7642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:01.350119  7642 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:01.351135  7642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:01.351441  7642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:01.352576  7642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:01.354395  7642 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:01.354508  7642 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:01.354626  7642 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:01.354703  7642 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:01.391906  7642 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:43861
I20240503 08:12:01.391958  7768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:43861 every 8 connection(s)
I20240503 08:12:01.393500  7642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:12:01.403749  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 7642
I20240503 08:12:01.403988  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:12:01.407603  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:01.408934  7769 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:01.409351  7769 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:01.410598  7769 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:01.412490  7452 ts_manager.cc:194] Registered new tserver with Master: 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:01.413800  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:44661
W20240503 08:12:01.759438  7772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:01.759784  7772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:01.759932  7772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:01.767755  7772 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:01.767910  7772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:01.768082  7772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:12:01.777571  7772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:01.778981  7772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:01.780733  7772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:01.794138  7777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.795037  7778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:01.795542  7780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:01.796595  7772 server_base.cc:1034] running on GCE node
I20240503 08:12:01.797377  7772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:01.798557  7772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:01.799826  7772 hybrid_clock.cc:648] HybridClock initialized: now 1714723921799771 us; error 52 us; skew 500 ppm
I20240503 08:12:01.800237  7772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:01.803102  7772 webserver.cc:460] Webserver started at http://127.2.49.131:42975/ using document root <none> and password file <none>
I20240503 08:12:01.803953  7772 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:01.804093  7772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:01.804436  7772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:01.807521  7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5d1ee23344854913811909afd87ab724"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-dz87"
I20240503 08:12:01.808411  7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5d1ee23344854913811909afd87ab724"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-dz87"
I20240503 08:12:01.814508  7772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20240503 08:12:01.818466  7786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:01.820228  7772 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:12:01.820447  7772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5d1ee23344854913811909afd87ab724"
format_stamp: "Formatted at 2024-05-03 08:12:01 on dist-test-slave-dz87"
I20240503 08:12:01.820670  7772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:01.845355  7772 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:01.846372  7772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:01.846746  7772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:01.847777  7772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:01.849395  7772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:01.849489  7772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:01.849578  7772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:01.849632  7772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:01.884145  7772 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:40273
I20240503 08:12:01.884276  7898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:40273 every 8 connection(s)
I20240503 08:12:01.885566  7772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:12:01.889953  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 7772
I20240503 08:12:01.890242  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:12:01.893886  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:01.903637  7899 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:01.903983  7899 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:01.904781  7899 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:01.906378  7452 ts_manager.cc:194] Registered new tserver with Master: 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:01.907205  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:57511
I20240503 08:12:01.944083  7638 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
W20240503 08:12:02.247596  7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:02.247921  7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:02.248044  7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:02.255723  7902 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:02.255868  7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:02.256042  7902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:12:02.265415  7902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:02.266847  7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:02.268637  7902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:02.281761  7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.282459  7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.282877  7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:02.284030  7902 server_base.cc:1034] running on GCE node
I20240503 08:12:02.284703  7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:02.285682  7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:02.286929  7902 hybrid_clock.cc:648] HybridClock initialized: now 1714723922286863 us; error 60 us; skew 500 ppm
I20240503 08:12:02.287322  7902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:02.290050  7902 webserver.cc:460] Webserver started at http://127.2.49.132:40655/ using document root <none> and password file <none>
I20240503 08:12:02.290887  7902 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:02.291025  7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:02.291359  7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:02.294299  7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fb22180119ec428393402ddbd0b748b6"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:02.295241  7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fb22180119ec428393402ddbd0b748b6"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:02.301090  7902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:12:02.305150  7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:02.307171  7902 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I20240503 08:12:02.307391  7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb22180119ec428393402ddbd0b748b6"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:02.307652  7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:02.333281  7902 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:02.334290  7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:02.334589  7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:02.335734  7902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:02.337414  7902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:02.337533  7902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:02.337666  7902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:02.337785  7902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:02.372980  7902 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:41615
I20240503 08:12:02.373035  8028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:41615 every 8 connection(s)
I20240503 08:12:02.374528  7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:12:02.376282  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 7902
I20240503 08:12:02.376451  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:12:02.379644  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:0
--local_ip_for_outbound_sockets=127.2.49.133
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:02.389050  8029 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:02.389371  8029 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:02.390228  8029 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:02.391696  7452 ts_manager.cc:194] Registered new tserver with Master: fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615)
I20240503 08:12:02.392561  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:35133
I20240503 08:12:02.417548  7769 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
W20240503 08:12:02.735309  8032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:02.735632  8032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:02.735752  8032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:02.743589  8032 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:02.743732  8032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:02.743897  8032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:12:02.753325  8032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:02.754755  8032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:02.756574  8032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:02.769436  8037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.770205  8040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:02.770941  8038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:02.771674  8032 server_base.cc:1034] running on GCE node
I20240503 08:12:02.772390  8032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:02.773406  8032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:02.774628  8032 hybrid_clock.cc:648] HybridClock initialized: now 1714723922774572 us; error 54 us; skew 500 ppm
I20240503 08:12:02.775090  8032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:02.788820  8032 webserver.cc:460] Webserver started at http://127.2.49.133:43589/ using document root <none> and password file <none>
I20240503 08:12:02.789690  8032 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:02.789825  8032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:02.790117  8032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:02.792913  8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1456c568b524465bb505956695d13738"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:02.793730  8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1456c568b524465bb505956695d13738"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:02.799407  8032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240503 08:12:02.803287  8046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:02.804797  8032 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:12:02.804991  8032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1456c568b524465bb505956695d13738"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:02.805224  8032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:02.827389  8032 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:02.828322  8032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:02.828622  8032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:02.829901  8032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:02.831686  8032 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:02.831784  8032 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:02.831905  8032 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:02.831972  8032 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:02.866524  8032 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:39615
I20240503 08:12:02.866557  8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:39615 every 8 connection(s)
I20240503 08:12:02.867982  8032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:02.872793  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 8032
I20240503 08:12:02.873030  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:12:02.876837  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.134:0
--local_ip_for_outbound_sockets=127.2.49.134
--webserver_interface=127.2.49.134
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:02.885083  8159 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:02.885481  8159 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:02.886359  8159 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:02.888041  7452 ts_manager.cc:194] Registered new tserver with Master: 1456c568b524465bb505956695d13738 (127.2.49.133:39615)
I20240503 08:12:02.889128  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:35071
I20240503 08:12:02.910259  7899 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
W20240503 08:12:03.234434  8162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:03.234771  8162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:03.234885  8162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:03.242858  8162 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:03.243007  8162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:03.243125  8162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.134
I20240503 08:12:03.252501  8162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.49.134
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.134
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:03.253839  8162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:03.255594  8162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:03.268410  8167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.269047  8170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.269997  8168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:03.270839  8162 server_base.cc:1034] running on GCE node
I20240503 08:12:03.271577  8162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:03.272641  8162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:03.273883  8162 hybrid_clock.cc:648] HybridClock initialized: now 1714723923273850 us; error 41 us; skew 500 ppm
I20240503 08:12:03.274358  8162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:03.276854  8162 webserver.cc:460] Webserver started at http://127.2.49.134:42933/ using document root <none> and password file <none>
I20240503 08:12:03.277657  8162 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:03.277786  8162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:03.278116  8162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:03.280977  8162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "b50f0dfea2284eacbe4184b1f3ae3753"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:03.281838  8162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "b50f0dfea2284eacbe4184b1f3ae3753"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:03.287595  8162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20240503 08:12:03.291515  8176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:03.293181  8162 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:12:03.293355  8162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b50f0dfea2284eacbe4184b1f3ae3753"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:03.293555  8162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:03.311604  8162 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:03.312517  8162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:03.312844  8162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:03.313889  8162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:03.315547  8162 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:03.315636  8162 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:03.315758  8162 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:03.315829  8162 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:03.347506  8162 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.134:37149
I20240503 08:12:03.347586  8288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.134:37149 every 8 connection(s)
I20240503 08:12:03.348934  8162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:03.351815  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 8162
I20240503 08:12:03.351994  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:12:03.355592  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.135:0
--local_ip_for_outbound_sockets=127.2.49.135
--webserver_interface=127.2.49.135
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:03.364197  8289 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:03.364521  8289 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:03.365403  8289 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:03.366892  7452 ts_manager.cc:194] Registered new tserver with Master: b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149)
I20240503 08:12:03.367698  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.134:34401
I20240503 08:12:03.395150  8029 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
W20240503 08:12:03.705266  8292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:03.705586  8292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:03.705726  8292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:03.713589  8292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:03.713749  8292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:03.713914  8292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.135
I20240503 08:12:03.723299  8292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.49.135
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.135
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:03.724705  8292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:03.726663  8292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:03.739528  8298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.740147  8297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:03.740913  8300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:03.741721  8292 server_base.cc:1034] running on GCE node
I20240503 08:12:03.742455  8292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:03.743464  8292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:03.744665  8292 hybrid_clock.cc:648] HybridClock initialized: now 1714723923744614 us; error 44 us; skew 500 ppm
I20240503 08:12:03.745000  8292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:03.747794  8292 webserver.cc:460] Webserver started at http://127.2.49.135:43435/ using document root <none> and password file <none>
I20240503 08:12:03.748580  8292 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:03.748698  8292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:03.749018  8292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:03.751808  8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "131e5ac1bcce424cbb1f7f60be20a263"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:03.752637  8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "131e5ac1bcce424cbb1f7f60be20a263"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:03.758277  8292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 08:12:03.762184  8306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:03.763731  8292 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:12:03.763935  8292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "131e5ac1bcce424cbb1f7f60be20a263"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:03.764170  8292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:03.799410  8292 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:03.800364  8292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:03.800645  8292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:03.801606  8292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:03.803138  8292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:03.803236  8292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:03.803350  8292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:03.803427  8292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:03.836025  8292 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.135:37243
I20240503 08:12:03.836086  8418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.135:37243 every 8 connection(s)
I20240503 08:12:03.837441  8292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:03.847394  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 8292
I20240503 08:12:03.847625  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:12:03.851169  8419 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:03.851495  8419 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:03.852246  8419 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:03.853652  7452 ts_manager.cc:194] Registered new tserver with Master: 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135:37243)
I20240503 08:12:03.854512  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.135:57325
I20240503 08:12:03.864615  2246 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:12:03.882225  7452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39096:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 08:12:03.892449  8159 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
I20240503 08:12:03.930672  7964 tablet_service.cc:1467] Processing CreateTablet for tablet 0246c726f68848a7ad4620e533f93b53 (DEFAULT_TABLE table=TestTable [id=7d99db23369f4ce98f11aa4d899291b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.930672  7573 tablet_service.cc:1467] Processing CreateTablet for tablet 0246c726f68848a7ad4620e533f93b53 (DEFAULT_TABLE table=TestTable [id=7d99db23369f4ce98f11aa4d899291b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.932394  7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0246c726f68848a7ad4620e533f93b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.932386  7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0246c726f68848a7ad4620e533f93b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.947327  8436 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:03.948305  8435 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Bootstrap starting.
I20240503 08:12:03.952188  8435 tablet_bootstrap.cc:654] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.952822  8436 tablet_bootstrap.cc:654] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.953553  8435 log.cc:824] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:03.954066  8436 log.cc:824] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:03.956336  8435 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: No bootstrap required, opened a new log
I20240503 08:12:03.956555  8436 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:03.956620  8435 ts_tablet_manager.cc:1367] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.007s
I20240503 08:12:03.956810  8436 ts_tablet_manager.cc:1367] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.000s
I20240503 08:12:03.963552  8435 raft_consensus.cc:348] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:03.963624  8436 raft_consensus.cc:348] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:03.963944  8435 raft_consensus.cc:374] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.963964  8436 raft_consensus.cc:374] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.964059  8436 raft_consensus.cc:729] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.964058  8435 raft_consensus.cc:729] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.964723  8435 consensus_queue.cc:260] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:03.964743  8436 consensus_queue.cc:260] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:03.967002  8435 ts_tablet_manager.cc:1398] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.002s
I20240503 08:12:03.967607  8436 ts_tablet_manager.cc:1398] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20240503 08:12:04.013199  8439 raft_consensus.cc:482] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:04.013533  8439 raft_consensus.cc:504] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:04.015139  8439 leader_election.cc:288] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615)
I20240503 08:12:04.021822  7984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0246c726f68848a7ad4620e533f93b53" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb22180119ec428393402ddbd0b748b6" is_pre_election: true
I20240503 08:12:04.022367  7984 raft_consensus.cc:2454] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:04.023164  7529 leader_election.cc:302] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, fb22180119ec428393402ddbd0b748b6; no voters: 
I20240503 08:12:04.023629  8439 raft_consensus.cc:2781] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:04.023774  8439 raft_consensus.cc:482] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.023880  8439 raft_consensus.cc:3037] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:04.026041  8439 raft_consensus.cc:504] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:04.026654  8439 leader_election.cc:288] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615)
I20240503 08:12:04.027333  7984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0246c726f68848a7ad4620e533f93b53" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb22180119ec428393402ddbd0b748b6"
I20240503 08:12:04.027539  7984 raft_consensus.cc:3037] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:04.029676  7984 raft_consensus.cc:2454] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:04.030247  7529 leader_election.cc:302] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, fb22180119ec428393402ddbd0b748b6; no voters: 
I20240503 08:12:04.030591  8439 raft_consensus.cc:2781] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:04.031113  8439 raft_consensus.cc:686] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:04.031572  8439 consensus_queue.cc:237] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:04.036512  7451 catalog_manager.cc:5484] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } }
I20240503 08:12:04.072489  2246 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:12:04.075018  2246 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 55fc889503854003801521bf97120179 to finish bootstrapping
I20240503 08:12:04.084412  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 359f2f323e79431eac07dc0a3f47a47b to finish bootstrapping
I20240503 08:12:04.093966  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5d1ee23344854913811909afd87ab724 to finish bootstrapping
I20240503 08:12:04.103747  2246 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fb22180119ec428393402ddbd0b748b6 to finish bootstrapping
I20240503 08:12:04.111644  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1456c568b524465bb505956695d13738 to finish bootstrapping
I20240503 08:12:04.121395  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b50f0dfea2284eacbe4184b1f3ae3753 to finish bootstrapping
W20240503 08:12:04.127377  8030 tablet.cc:2369] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:04.130591  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 131e5ac1bcce424cbb1f7f60be20a263 to finish bootstrapping
I20240503 08:12:04.140322  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 7902
I20240503 08:12:04.141446  8024 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:12:04.173327  7639 tablet.cc:2369] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:04.233912  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 7902
I20240503 08:12:04.251286  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 8032
I20240503 08:12:04.252468  8154 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:04.335065  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 8032
I20240503 08:12:04.355558  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 8162
I20240503 08:12:04.356674  8284 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:04.438649  8289 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
I20240503 08:12:04.439527  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 8162
W20240503 08:12:04.455868  7439 connection.cc:535] server connection from 127.2.49.134:34401 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:04.456032  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 8292
I20240503 08:12:04.457208  8414 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:12:04.459501  7529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111)
W20240503 08:12:04.461444  7529 consensus_peers.cc:485] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:04.551359  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 8292
W20240503 08:12:06.943892  7529 consensus_peers.cc:485] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:12:08.324281  7451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39096:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:12:08.336616  7573 tablet_service.cc:1467] Processing CreateTablet for tablet 0563aa33bb13450c96f620998b881c61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.337309  7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0563aa33bb13450c96f620998b881c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.351364  8475 tablet_bootstrap.cc:492] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:08.353504  8475 tablet_bootstrap.cc:654] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.356324  7704 tablet_service.cc:1467] Processing CreateTablet for tablet 6462c70bafc94b7fb3bcb9502e80e096 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.357179  7702 tablet_service.cc:1467] Processing CreateTablet for tablet 0563aa33bb13450c96f620998b881c61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.356498  7703 tablet_service.cc:1467] Processing CreateTablet for tablet a629b67231f84cdbb5e302265896e25a (DEFAULT_TABLE table=rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.358222  7704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6462c70bafc94b7fb3bcb9502e80e096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.358796  7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0563aa33bb13450c96f620998b881c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.360522  7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a629b67231f84cdbb5e302265896e25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.363345  8475 tablet_bootstrap.cc:492] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:08.363528  8475 ts_tablet_manager.cc:1367] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240503 08:12:08.366593  8475 raft_consensus.cc:348] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.373348  8475 raft_consensus.cc:374] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.373495  8475 raft_consensus.cc:729] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.372848  7834 tablet_service.cc:1467] Processing CreateTablet for tablet 6462c70bafc94b7fb3bcb9502e80e096 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.372900  7833 tablet_service.cc:1467] Processing CreateTablet for tablet a629b67231f84cdbb5e302265896e25a (DEFAULT_TABLE table=rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.374565  8475 consensus_queue.cc:260] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.375751  8475 ts_tablet_manager.cc:1398] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
I20240503 08:12:08.376356  7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a629b67231f84cdbb5e302265896e25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.377301  8477 tablet_bootstrap.cc:492] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:08.378130  7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6462c70bafc94b7fb3bcb9502e80e096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.380975  8477 tablet_bootstrap.cc:654] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.382086  8477 log.cc:824] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:08.384821  8477 tablet_bootstrap.cc:492] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:08.385121  8477 ts_tablet_manager.cc:1367] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240503 08:12:08.391001  8479 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:08.393998  8477 raft_consensus.cc:348] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.394481  8477 raft_consensus.cc:374] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.394603  8477 raft_consensus.cc:729] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.395197  8479 tablet_bootstrap.cc:654] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.395263  8477 consensus_queue.cc:260] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.396519  8479 log.cc:824] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:08.398481  8477 ts_tablet_manager.cc:1398] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.013s	user 0.013s	sys 0.000s
I20240503 08:12:08.398957  8477 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:08.399999  8479 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:08.400328  8479 ts_tablet_manager.cc:1367] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240503 08:12:08.401037  8477 tablet_bootstrap.cc:654] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:08.401453  7770 tablet.cc:2369] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:08.404814  8477 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:08.404989  8477 ts_tablet_manager.cc:1367] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.003s
I20240503 08:12:08.406025  8477 raft_consensus.cc:348] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.406282  8477 raft_consensus.cc:374] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.406379  8477 raft_consensus.cc:729] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.406646  8477 consensus_queue.cc:260] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.407269  8477 ts_tablet_manager.cc:1398] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:12:08.407644  8477 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:08.409452  8477 tablet_bootstrap.cc:654] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.409833  8479 raft_consensus.cc:348] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.410363  8479 raft_consensus.cc:374] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.410477  8479 raft_consensus.cc:729] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.411518  8479 consensus_queue.cc:260] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.416651  8477 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:08.416841  8477 ts_tablet_manager.cc:1367] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240503 08:12:08.417452  8477 raft_consensus.cc:348] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.417634  8477 raft_consensus.cc:374] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.417719  8477 raft_consensus.cc:729] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.418000  8477 consensus_queue.cc:260] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.418604  8477 ts_tablet_manager.cc:1398] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:12:08.418915  8479 ts_tablet_manager.cc:1398] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.018s	user 0.013s	sys 0.004s
I20240503 08:12:08.419272  8479 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:08.420943  8479 tablet_bootstrap.cc:654] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.422967  8479 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:08.423127  8479 ts_tablet_manager.cc:1367] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:12:08.423944  8479 raft_consensus.cc:348] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.424118  8479 raft_consensus.cc:374] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.424216  8479 raft_consensus.cc:729] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.424469  8479 consensus_queue.cc:260] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.424933  8479 ts_tablet_manager.cc:1398] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240503 08:12:08.645453  7900 tablet.cc:2369] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:08.742969  8464 raft_consensus.cc:482] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.743181  8464 raft_consensus.cc:504] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.743870  8464 leader_election.cc:288] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:08.749837  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0563aa33bb13450c96f620998b881c61" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:08.750494  7724 raft_consensus.cc:2454] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:08.751163  7529 leader_election.cc:302] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:08.751492  8464 raft_consensus.cc:2781] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.751616  8464 raft_consensus.cc:482] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.751716  8464 raft_consensus.cc:3037] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.753635  8464 raft_consensus.cc:504] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.754189  8464 leader_election.cc:288] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:08.754869  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0563aa33bb13450c96f620998b881c61" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:08.755069  7724 raft_consensus.cc:3037] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.757277  7724 raft_consensus.cc:2454] T 0563aa33bb13450c96f620998b881c61 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:08.757781  7529 leader_election.cc:302] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:08.758119  8464 raft_consensus.cc:2781] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:08.758596  8464 raft_consensus.cc:686] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:08.758908  8464 consensus_queue.cc:237] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:08.761288  7452 catalog_manager.cc:5484] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } }
I20240503 08:12:08.785212  8485 raft_consensus.cc:482] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.785492  8485 raft_consensus.cc:504] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.786612  8490 raft_consensus.cc:482] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.786782  8490 raft_consensus.cc:504] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.787292  8485 leader_election.cc:288] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:08.787397  8490 leader_election.cc:288] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:08.793551  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a629b67231f84cdbb5e302265896e25a" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:08.793644  7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6462c70bafc94b7fb3bcb9502e80e096" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:08.793864  7724 raft_consensus.cc:2454] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 0.
I20240503 08:12:08.794003  7723 raft_consensus.cc:2454] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 0.
I20240503 08:12:08.794464  7790 leader_election.cc:302] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:08.794881  8490 raft_consensus.cc:2781] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.795025  8490 raft_consensus.cc:482] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.795136  8490 raft_consensus.cc:3037] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.795087  7790 leader_election.cc:302] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:08.795462  8485 raft_consensus.cc:2781] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.795611  8485 raft_consensus.cc:482] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.795706  8485 raft_consensus.cc:3037] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.797660  8490 raft_consensus.cc:504] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.797742  8485 raft_consensus.cc:504] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.798254  8490 leader_election.cc:288] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:08.798286  8485 leader_election.cc:288] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:08.798883  7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a629b67231f84cdbb5e302265896e25a" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:08.798995  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6462c70bafc94b7fb3bcb9502e80e096" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:08.799096  7723 raft_consensus.cc:3037] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.799232  7724 raft_consensus.cc:3037] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.801326  7724 raft_consensus.cc:2454] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 1.
I20240503 08:12:08.801429  7723 raft_consensus.cc:2454] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 1.
I20240503 08:12:08.801826  7790 leader_election.cc:302] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:08.802141  8485 raft_consensus.cc:2781] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:08.802281  7790 leader_election.cc:302] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:08.802346  8485 raft_consensus.cc:686] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:12:08.802843  8490 raft_consensus.cc:2781] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:08.802889  8485 consensus_queue.cc:237] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:08.803339  8490 raft_consensus.cc:686] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:12:08.803745  8490 consensus_queue.cc:237] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:08.807931  7452 catalog_manager.cc:5484] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1ee23344854913811909afd87ab724 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
I20240503 08:12:08.813002  7452 catalog_manager.cc:5484] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1ee23344854913811909afd87ab724 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } }
W20240503 08:12:09.180986  8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:09.181377  8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:09.189046  8495 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
W20240503 08:12:09.228430  7529 consensus_peers.cc:485] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:12:09.245972  8485 consensus_queue.cc:1035] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.276795  8490 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.300235  8464 consensus_queue.cc:1035] T 0563aa33bb13450c96f620998b881c61 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 247.988 ms
  time per row: 0.123994 ms
I20240503 08:12:09.590682  7450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39096:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:12:09.601512  7573 tablet_service.cc:1467] Processing CreateTablet for tablet 303fded976c342efbccfc375950f78ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.601723  7833 tablet_service.cc:1467] Processing CreateTablet for tablet 303fded976c342efbccfc375950f78ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.602051  7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303fded976c342efbccfc375950f78ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.602123  7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303fded976c342efbccfc375950f78ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.602056  7834 tablet_service.cc:1467] Processing CreateTablet for tablet 8967d6dac1aa482683c8bad1892dcbde (DEFAULT_TABLE table=rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.602969  7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8967d6dac1aa482683c8bad1892dcbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.602929  7702 tablet_service.cc:1467] Processing CreateTablet for tablet a7f1cb831a9149e494cb31fac27f922d (DEFAULT_TABLE table=rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.603507  7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f1cb831a9149e494cb31fac27f922d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.607277  7572 tablet_service.cc:1467] Processing CreateTablet for tablet 8967d6dac1aa482683c8bad1892dcbde (DEFAULT_TABLE table=rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.607952  7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8967d6dac1aa482683c8bad1892dcbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.611562  7571 tablet_service.cc:1467] Processing CreateTablet for tablet a7f1cb831a9149e494cb31fac27f922d (DEFAULT_TABLE table=rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.615953  8554 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:09.616860  8553 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:09.618853  8555 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:09.619009  8553 tablet_bootstrap.cc:654] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.619164  7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f1cb831a9149e494cb31fac27f922d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.622781  8554 tablet_bootstrap.cc:654] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.623358  8553 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:09.623520  8553 ts_tablet_manager.cc:1367] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 08:12:09.624339  8553 raft_consensus.cc:348] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.624562  8553 raft_consensus.cc:374] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.624660  8553 raft_consensus.cc:729] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.624882  8553 consensus_queue.cc:260] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.625671  8555 tablet_bootstrap.cc:654] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.625665  8553 ts_tablet_manager.cc:1398] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:12:09.625945  8553 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:09.627569  8553 tablet_bootstrap.cc:654] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.628988  8555 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:09.629150  8555 ts_tablet_manager.cc:1367] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240503 08:12:09.630321  8554 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:09.630484  8554 ts_tablet_manager.cc:1367] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.001s
I20240503 08:12:09.630689  8553 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:09.630792  8555 raft_consensus.cc:348] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:09.630913  8553 ts_tablet_manager.cc:1367] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:12:09.631006  8555 raft_consensus.cc:374] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.631107  8555 raft_consensus.cc:729] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.631301  8554 raft_consensus.cc:348] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.631394  8555 consensus_queue.cc:260] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:09.631516  8554 raft_consensus.cc:374] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.631608  8554 raft_consensus.cc:729] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.631810  8553 raft_consensus.cc:348] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.631835  8554 consensus_queue.cc:260] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.632004  8553 raft_consensus.cc:374] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.632084  8553 raft_consensus.cc:729] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.632409  8554 ts_tablet_manager.cc:1398] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:12:09.632445  8553 consensus_queue.cc:260] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.632710  8554 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:09.632817  8555 ts_tablet_manager.cc:1398] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:12:09.632992  8553 ts_tablet_manager.cc:1398] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:12:09.634546  8554 tablet_bootstrap.cc:654] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.637856  8554 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:09.638046  8554 ts_tablet_manager.cc:1367] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 08:12:09.638896  8554 raft_consensus.cc:348] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.639070  8554 raft_consensus.cc:374] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.639118  8554 raft_consensus.cc:729] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.639333  8554 consensus_queue.cc:260] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.639792  8554 ts_tablet_manager.cc:1398] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 08:12:09.640122  8554 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:09.641484  8554 tablet_bootstrap.cc:654] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:09.643416  8554 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:09.643538  8554 ts_tablet_manager.cc:1367] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:12:09.644268  8554 raft_consensus.cc:348] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:09.644486  8554 raft_consensus.cc:374] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:09.644587  8554 raft_consensus.cc:729] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:09.644811  8554 consensus_queue.cc:260] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:09.645303  8554 ts_tablet_manager.cc:1398] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240503 08:12:09.646570  7529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:12:09.651896  8485 raft_consensus.cc:482] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.652101  8485 raft_consensus.cc:504] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.652657  8485 leader_election.cc:288] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:09.659140  7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8967d6dac1aa482683c8bad1892dcbde" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc889503854003801521bf97120179" is_pre_election: true
I20240503 08:12:09.659538  7593 raft_consensus.cc:2454] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 0.
I20240503 08:12:09.660046  7789 leader_election.cc:302] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:09.660372  8485 raft_consensus.cc:2781] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.660459  8485 raft_consensus.cc:482] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.660558  8485 raft_consensus.cc:3037] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.662612  8485 raft_consensus.cc:504] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.663077  8485 leader_election.cc:288] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Requested vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:09.663635  7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8967d6dac1aa482683c8bad1892dcbde" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:12:09.663823  7593 raft_consensus.cc:3037] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.666020  7593 raft_consensus.cc:2454] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 1.
I20240503 08:12:09.666451  7789 leader_election.cc:302] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:09.666761  8485 raft_consensus.cc:2781] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.666925  8485 raft_consensus.cc:686] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:12:09.667173  8485 consensus_queue.cc:237] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.669555  7448 catalog_manager.cc:5484] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1ee23344854913811909afd87ab724 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
I20240503 08:12:09.927842  8485 raft_consensus.cc:482] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.928073  8485 raft_consensus.cc:504] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.928673  8485 leader_election.cc:288] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:09.929293  7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303fded976c342efbccfc375950f78ec" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc889503854003801521bf97120179" is_pre_election: true
I20240503 08:12:09.929534  7593 raft_consensus.cc:2454] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 0.
I20240503 08:12:09.929941  7789 leader_election.cc:302] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:09.930253  8485 raft_consensus.cc:2781] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.930382  8485 raft_consensus.cc:482] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.930471  8485 raft_consensus.cc:3037] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.932335  8485 raft_consensus.cc:504] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.932863  8485 leader_election.cc:288] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Requested vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:09.933439  7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303fded976c342efbccfc375950f78ec" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:12:09.933655  7593 raft_consensus.cc:3037] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.935660  7593 raft_consensus.cc:2454] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 1.
I20240503 08:12:09.936126  7789 leader_election.cc:302] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:09.936448  8485 raft_consensus.cc:2781] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.936573  8485 raft_consensus.cc:686] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:12:09.936861  8485 consensus_queue.cc:237] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:09.939342  7448 catalog_manager.cc:5484] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1ee23344854913811909afd87ab724 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
I20240503 08:12:10.076316  8464 raft_consensus.cc:482] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:10.076545  8464 raft_consensus.cc:504] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:10.077172  8464 leader_election.cc:288] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:10.077869  7722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f1cb831a9149e494cb31fac27f922d" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:10.078145  7722 raft_consensus.cc:2454] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:10.078737  7529 leader_election.cc:302] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:10.079408  8464 raft_consensus.cc:2781] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:10.079746  8464 raft_consensus.cc:482] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.079876  8464 raft_consensus.cc:3037] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.082192  8464 raft_consensus.cc:504] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:10.082760  8464 leader_election.cc:288] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:10.083415  7722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f1cb831a9149e494cb31fac27f922d" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:10.083632  7722 raft_consensus.cc:3037] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.085506  7722 raft_consensus.cc:2454] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:10.086032  7529 leader_election.cc:302] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:10.086462  8464 raft_consensus.cc:2781] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:10.087023  8464 raft_consensus.cc:686] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:10.087306  8464 consensus_queue.cc:237] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:10.089912  7448 catalog_manager.cc:5484] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } }
I20240503 08:12:10.157737  8485 consensus_queue.cc:1035] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.323323  8543 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:10.535949  8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:10.536309  8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:10.544150  8570 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:10.545123  8464 consensus_queue.cc:1035] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 179.725 ms
  time per row: 0.0898625 ms
I20240503 08:12:10.878209  7447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39096:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:12:10.888042  7702 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5a34b052e94b7b8a1dbaec2038d921 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:10.888160  7834 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5a34b052e94b7b8a1dbaec2038d921 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:10.888603  7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5a34b052e94b7b8a1dbaec2038d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.888619  7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5a34b052e94b7b8a1dbaec2038d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.889066  7571 tablet_service.cc:1467] Processing CreateTablet for tablet 3242894c953a4f6f8095bfb8c9318855 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:10.889484  7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3242894c953a4f6f8095bfb8c9318855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.895140  7833 tablet_service.cc:1467] Processing CreateTablet for tablet 01f87cf2ca7041cd8727e41d994f330c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:10.895591  7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f87cf2ca7041cd8727e41d994f330c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.896797  7704 tablet_service.cc:1467] Processing CreateTablet for tablet 01f87cf2ca7041cd8727e41d994f330c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:10.896827  7703 tablet_service.cc:1467] Processing CreateTablet for tablet 3242894c953a4f6f8095bfb8c9318855 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:10.897367  7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3242894c953a4f6f8095bfb8c9318855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.898528  7704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f87cf2ca7041cd8727e41d994f330c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:10.907028  8625 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:10.909688  8625 tablet_bootstrap.cc:654] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.909955  8626 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:10.912549  8626 tablet_bootstrap.cc:654] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.913331  8627 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:10.915431  8627 tablet_bootstrap.cc:654] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.918663  8625 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:10.918884  8625 ts_tablet_manager.cc:1367] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.000s
I20240503 08:12:10.919978  8625 raft_consensus.cc:348] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.920257  8625 raft_consensus.cc:374] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.920364  8625 raft_consensus.cc:729] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.920755  8625 consensus_queue.cc:260] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.921658  8625 ts_tablet_manager.cc:1398] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:12:10.922901  8627 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:10.923100  8627 ts_tablet_manager.cc:1367] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240503 08:12:10.923704  8625 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:10.923861  8627 raft_consensus.cc:348] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.924075  8627 raft_consensus.cc:374] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.924172  8627 raft_consensus.cc:729] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.924407  8627 consensus_queue.cc:260] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.924890  8627 ts_tablet_manager.cc:1398] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:12:10.925143  8627 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:10.926369  8625 tablet_bootstrap.cc:654] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.926963  8627 tablet_bootstrap.cc:654] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.931162  8626 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:10.931362  8626 ts_tablet_manager.cc:1367] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.022s	user 0.002s	sys 0.004s
I20240503 08:12:10.932185  8626 raft_consensus.cc:348] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.932415  8626 raft_consensus.cc:374] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.932520  8626 raft_consensus.cc:729] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.932754  8626 consensus_queue.cc:260] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.933349  8626 ts_tablet_manager.cc:1398] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:12:10.934469  8627 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:10.934664  8627 ts_tablet_manager.cc:1367] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.003s
I20240503 08:12:10.935213  8625 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:10.935384  8625 ts_tablet_manager.cc:1367] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.007s
I20240503 08:12:10.935540  8627 raft_consensus.cc:348] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.935771  8627 raft_consensus.cc:374] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.935844  8627 raft_consensus.cc:729] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.936089  8627 consensus_queue.cc:260] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.936101  8625 raft_consensus.cc:348] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.936328  8625 raft_consensus.cc:374] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.936407  8625 raft_consensus.cc:729] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.936657  8627 ts_tablet_manager.cc:1398] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:12:10.936631  8625 consensus_queue.cc:260] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.937152  8625 ts_tablet_manager.cc:1398] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:12:10.942577  8625 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:10.944286  8625 tablet_bootstrap.cc:654] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.947355  8625 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:10.947527  8625 ts_tablet_manager.cc:1367] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240503 08:12:10.948279  8625 raft_consensus.cc:348] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.948480  8625 raft_consensus.cc:374] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.948554  8625 raft_consensus.cc:729] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.948786  8625 consensus_queue.cc:260] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:10.949308  8625 ts_tablet_manager.cc:1398] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:12:10.960538  8575 raft_consensus.cc:482] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:10.960762  8575 raft_consensus.cc:504] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.962218  8575 leader_election.cc:288] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:10.968446  7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3242894c953a4f6f8095bfb8c9318855" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc889503854003801521bf97120179" is_pre_election: true
I20240503 08:12:10.968703  7592 raft_consensus.cc:2454] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 0.
I20240503 08:12:10.969321  7659 leader_election.cc:302] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:10.969807  8575 raft_consensus.cc:2781] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:10.969939  8575 raft_consensus.cc:482] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.970063  8575 raft_consensus.cc:3037] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.972149  8575 raft_consensus.cc:504] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.972690  8575 leader_election.cc:288] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 election: Requested vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:10.973326  7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3242894c953a4f6f8095bfb8c9318855" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:12:10.973524  7592 raft_consensus.cc:3037] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.975621  7592 raft_consensus.cc:2454] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 1.
I20240503 08:12:10.976140  7659 leader_election.cc:302] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:10.976511  8575 raft_consensus.cc:2781] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:10.976950  8575 raft_consensus.cc:686] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Becoming Leader. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Running, Role: LEADER
I20240503 08:12:10.977455  8575 consensus_queue.cc:237] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:10.980600  7449 catalog_manager.cc:5484] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: term changed from 0 to 1, leader changed from <none> to 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130). New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } }
I20240503 08:12:11.048224  8610 raft_consensus.cc:482] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.048430  8610 raft_consensus.cc:504] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:11.049036  8610 leader_election.cc:288] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:11.049765  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5a34b052e94b7b8a1dbaec2038d921" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:11.049986  7724 raft_consensus.cc:2454] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 0.
I20240503 08:12:11.050460  7790 leader_election.cc:302] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:11.050741  8610 raft_consensus.cc:2781] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.050858  8610 raft_consensus.cc:482] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.050951  8610 raft_consensus.cc:3037] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.052897  8610 raft_consensus.cc:504] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:11.053400  8610 leader_election.cc:288] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:11.054028  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5a34b052e94b7b8a1dbaec2038d921" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:11.054250  7724 raft_consensus.cc:3037] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.056310  7724 raft_consensus.cc:2454] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 1.
I20240503 08:12:11.056788  7790 leader_election.cc:302] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:11.057118  8610 raft_consensus.cc:2781] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.057287  8610 raft_consensus.cc:686] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:12:11.057557  8610 consensus_queue.cc:237] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:11.060317  7449 catalog_manager.cc:5484] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1ee23344854913811909afd87ab724 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
I20240503 08:12:11.179849  8575 raft_consensus.cc:482] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.180073  8575 raft_consensus.cc:504] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:11.180708  8575 leader_election.cc:288] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:11.186571  7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f87cf2ca7041cd8727e41d994f330c" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724" is_pre_election: true
I20240503 08:12:11.186996  7854 raft_consensus.cc:2454] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 0.
I20240503 08:12:11.187546  7658 leader_election.cc:302] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:11.187954  8575 raft_consensus.cc:2781] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.188071  8575 raft_consensus.cc:482] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.188179  8575 raft_consensus.cc:3037] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.190243  8575 raft_consensus.cc:504] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:11.190804  8575 leader_election.cc:288] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:11.191443  7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f87cf2ca7041cd8727e41d994f330c" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:12:11.191637  7854 raft_consensus.cc:3037] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.193925  7854 raft_consensus.cc:2454] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 1.
I20240503 08:12:11.194543  7658 leader_election.cc:302] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:11.194867  8575 raft_consensus.cc:2781] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.195061  8575 raft_consensus.cc:686] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Becoming Leader. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Running, Role: LEADER
I20240503 08:12:11.195334  8575 consensus_queue.cc:237] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:11.198084  7449 catalog_manager.cc:5484] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: term changed from 0 to 1, leader changed from <none> to 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130). New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
I20240503 08:12:11.495889  8575 consensus_queue.cc:1035] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:11.532192  7529 consensus_peers.cc:485] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:12:11.623675  8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:11.624032  8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:11.631736  8641 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:11.654732  8610 consensus_queue.cc:1035] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:12:11.715358  8635 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 204.638 ms
  time per row: 0.102319 ms
I20240503 08:12:11.996482  7450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39096:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:12:12.005676  7571 tablet_service.cc:1467] Processing CreateTablet for tablet 6c6b0211f6b44c1b92a5be884608d000 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.005839  7833 tablet_service.cc:1467] Processing CreateTablet for tablet 6c6b0211f6b44c1b92a5be884608d000 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.005887  7834 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5a650ed26a406baa15105ac9fc8f8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.006224  7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c6b0211f6b44c1b92a5be884608d000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.006321  7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c6b0211f6b44c1b92a5be884608d000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.006909  7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5a650ed26a406baa15105ac9fc8f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.007274  7704 tablet_service.cc:1467] Processing CreateTablet for tablet 80eccc47e060493ebe43b007c7a23f00 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.007689  7704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eccc47e060493ebe43b007c7a23f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.008349  7573 tablet_service.cc:1467] Processing CreateTablet for tablet 80eccc47e060493ebe43b007c7a23f00 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.008733  7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eccc47e060493ebe43b007c7a23f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.011996  7572 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5a650ed26a406baa15105ac9fc8f8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.012428  7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5a650ed26a406baa15105ac9fc8f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.019812  8699 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:12.020226  8698 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:12.021647  8697 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:12.021833  8699 tablet_bootstrap.cc:654] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.022482  8698 tablet_bootstrap.cc:654] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.023576  8697 tablet_bootstrap.cc:654] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.030838  8699 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:12.030975  8697 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:12.031018  8699 ts_tablet_manager.cc:1367] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240503 08:12:12.031148  8697 ts_tablet_manager.cc:1367] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240503 08:12:12.031181  8698 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:12.031322  8698 ts_tablet_manager.cc:1367] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240503 08:12:12.031872  8699 raft_consensus.cc:348] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.032092  8699 raft_consensus.cc:374] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.031982  8697 raft_consensus.cc:348] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.032183  8699 raft_consensus.cc:729] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.032088  8698 raft_consensus.cc:348] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.032193  8697 raft_consensus.cc:374] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.032271  8698 raft_consensus.cc:374] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.032290  8697 raft_consensus.cc:729] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.032358  8698 raft_consensus.cc:729] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.032409  8699 consensus_queue.cc:260] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.032562  8697 consensus_queue.cc:260] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.032586  8698 consensus_queue.cc:260] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.033421  8698 ts_tablet_manager.cc:1398] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:12:12.033659  8697 ts_tablet_manager.cc:1398] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:12:12.033787  8699 ts_tablet_manager.cc:1398] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:12:12.034082  8699 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:12.034389  8697 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:12.035934  8697 tablet_bootstrap.cc:654] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.036130  8699 tablet_bootstrap.cc:654] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.038483  8697 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:12.038744  8697 ts_tablet_manager.cc:1367] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:12:12.039433  8699 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:12.039669  8699 ts_tablet_manager.cc:1367] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 08:12:12.039743  8697 raft_consensus.cc:348] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.039995  8697 raft_consensus.cc:374] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.040629  8699 raft_consensus.cc:348] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.040092  8697 raft_consensus.cc:729] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.040843  8699 raft_consensus.cc:374] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.040925  8699 raft_consensus.cc:729] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.040971  8697 consensus_queue.cc:260] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.041172  8699 consensus_queue.cc:260] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.041545  8697 ts_tablet_manager.cc:1398] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:12:12.041805  8699 ts_tablet_manager.cc:1398] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:12:12.042107  8699 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:12.043802  8699 tablet_bootstrap.cc:654] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.045737  8699 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:12.045917  8699 ts_tablet_manager.cc:1367] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 08:12:12.046597  8699 raft_consensus.cc:348] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.046772  8699 raft_consensus.cc:374] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.046866  8699 raft_consensus.cc:729] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.047092  8699 consensus_queue.cc:260] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.047600  8699 ts_tablet_manager.cc:1398] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:12:12.254067  8565 raft_consensus.cc:482] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.254312  8565 raft_consensus.cc:504] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.255021  8565 leader_election.cc:288] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:12.258813  8565 raft_consensus.cc:482] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.259034  8565 raft_consensus.cc:504] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.259618  8565 leader_election.cc:288] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:12.260213  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:12.260458  7724 raft_consensus.cc:2454] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:12.260954  7529 leader_election.cc:302] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:12.261022  7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5a650ed26a406baa15105ac9fc8f8f" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724" is_pre_election: true
I20240503 08:12:12.261255  7854 raft_consensus.cc:2454] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:12.261363  8565 raft_consensus.cc:2781] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.261478  8565 raft_consensus.cc:482] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.261610  8565 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.261723  7527 leader_election.cc:302] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:12.262601  8706 raft_consensus.cc:2781] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.262714  8706 raft_consensus.cc:482] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.262866  8706 raft_consensus.cc:3037] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.264055  8565 raft_consensus.cc:504] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.264624  8565 leader_election.cc:288] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:12.265048  8706 raft_consensus.cc:504] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.265233  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:12.265471  7724 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.265588  8706 leader_election.cc:288] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:12.266144  7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5a650ed26a406baa15105ac9fc8f8f" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:12:12.266412  7854 raft_consensus.cc:3037] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.268262  7724 raft_consensus.cc:2454] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:12.268775  7529 leader_election.cc:302] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:12.268956  7854 raft_consensus.cc:2454] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:12.269124  8706 raft_consensus.cc:2781] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.269448  8706 raft_consensus.cc:686] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:12.269502  7527 leader_election.cc:302] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:12.269811  8565 raft_consensus.cc:2781] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.269752  8706 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.270401  8565 raft_consensus.cc:686] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:12.270665  8565 consensus_queue.cc:237] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:12.273077  7447 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } }
I20240503 08:12:12.277678  7447 catalog_manager.cc:5484] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } }
I20240503 08:12:12.327236  8565 raft_consensus.cc:482] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.327574  8565 raft_consensus.cc:504] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.328176  8565 leader_election.cc:288] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:12.328838  7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c6b0211f6b44c1b92a5be884608d000" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724" is_pre_election: true
I20240503 08:12:12.329056  7854 raft_consensus.cc:2454] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:12.329511  7527 leader_election.cc:302] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:12.329808  8565 raft_consensus.cc:2781] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.329910  8565 raft_consensus.cc:482] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.330024  8565 raft_consensus.cc:3037] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.331983  8565 raft_consensus.cc:504] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.332473  8565 leader_election.cc:288] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:12.333091  7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c6b0211f6b44c1b92a5be884608d000" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:12:12.333277  7854 raft_consensus.cc:3037] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.335301  7854 raft_consensus.cc:2454] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:12.335803  7527 leader_election.cc:302] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:12.336120  8565 raft_consensus.cc:2781] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.336248  8565 raft_consensus.cc:686] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:12.336498  8565 consensus_queue.cc:237] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:12.339435  7447 catalog_manager.cc:5484] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
W20240503 08:12:12.744583  8711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:12.744927  8711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:12.752542  8711 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:12.764557  8565 consensus_queue.cc:1035] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 08:12:12.832938  8706 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.896883  8706 consensus_queue.cc:1035] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 197.477 ms
  time per row: 0.0987387 ms
I20240503 08:12:13.096267  7447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39096:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:12:13.105562  7834 tablet_service.cc:1467] Processing CreateTablet for tablet 44c94e75cff640f8927043b5e071c980 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.105780  7704 tablet_service.cc:1467] Processing CreateTablet for tablet 44c94e75cff640f8927043b5e071c980 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.106065  7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c94e75cff640f8927043b5e071c980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.106565  7571 tablet_service.cc:1467] Processing CreateTablet for tablet 65167136cc2d478cb0efd6a374839c48 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.106941  7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65167136cc2d478cb0efd6a374839c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.107812  7833 tablet_service.cc:1467] Processing CreateTablet for tablet 91ae2ad1ed4448689d8fe8eba7942dd9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.108157  7703 tablet_service.cc:1467] Processing CreateTablet for tablet 91ae2ad1ed4448689d8fe8eba7942dd9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.108281  7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ae2ad1ed4448689d8fe8eba7942dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.108814  7702 tablet_service.cc:1467] Processing CreateTablet for tablet 65167136cc2d478cb0efd6a374839c48 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.109441  7704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c94e75cff640f8927043b5e071c980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.114447  7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65167136cc2d478cb0efd6a374839c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.118367  7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ae2ad1ed4448689d8fe8eba7942dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.118750  8763 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:13.121342  8763 tablet_bootstrap.cc:654] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.123740  8764 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: Bootstrap starting.
I20240503 08:12:13.126474  8764 tablet_bootstrap.cc:654] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.130368  8765 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:13.130604  8763 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:13.130683  8764 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: No bootstrap required, opened a new log
I20240503 08:12:13.130764  8763 ts_tablet_manager.cc:1367] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.006s
I20240503 08:12:13.130844  8764 ts_tablet_manager.cc:1367] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 08:12:13.131654  8764 raft_consensus.cc:348] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.131646  8763 raft_consensus.cc:348] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.131836  8764 raft_consensus.cc:374] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.131845  8763 raft_consensus.cc:374] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.131932  8763 raft_consensus.cc:729] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.131983  8764 raft_consensus.cc:729] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.132316  8765 tablet_bootstrap.cc:654] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.132236  8764 consensus_queue.cc:260] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.132189  8763 consensus_queue.cc:260] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.132776  8764 ts_tablet_manager.cc:1398] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:12:13.132959  8763 ts_tablet_manager.cc:1398] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:12:13.133256  8763 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: Bootstrap starting.
I20240503 08:12:13.135592  8763 tablet_bootstrap.cc:654] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.137933  8763 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: No bootstrap required, opened a new log
I20240503 08:12:13.138099  8763 ts_tablet_manager.cc:1367] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:12:13.138120  8765 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:13.138386  8765 ts_tablet_manager.cc:1367] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240503 08:12:13.138936  8763 raft_consensus.cc:348] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.139238  8763 raft_consensus.cc:374] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.139238  8765 raft_consensus.cc:348] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.139377  8763 raft_consensus.cc:729] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.139461  8765 raft_consensus.cc:374] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.139562  8765 raft_consensus.cc:729] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.139633  8763 consensus_queue.cc:260] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.139770  8765 consensus_queue.cc:260] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.140383  8765 ts_tablet_manager.cc:1398] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:12:13.140702  8765 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:13.142464  8765 tablet_bootstrap.cc:654] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.140712  8763 ts_tablet_manager.cc:1398] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:12:13.144460  8765 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:13.144659  8765 ts_tablet_manager.cc:1367] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:12:13.145480  8765 raft_consensus.cc:348] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.145658  8765 raft_consensus.cc:374] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.145714  8765 raft_consensus.cc:729] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.145907  8765 consensus_queue.cc:260] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.146373  8765 ts_tablet_manager.cc:1398] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:12:13.146603  8765 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b: Bootstrap starting.
I20240503 08:12:13.148358  8765 tablet_bootstrap.cc:654] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.150230  8765 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b: No bootstrap required, opened a new log
I20240503 08:12:13.150362  8765 ts_tablet_manager.cc:1367] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240503 08:12:13.151043  8765 raft_consensus.cc:348] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.151191  8765 raft_consensus.cc:374] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.151279  8765 raft_consensus.cc:729] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.151477  8765 consensus_queue.cc:260] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.151975  8765 ts_tablet_manager.cc:1398] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:12:13.167439  8750 raft_consensus.cc:482] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:13.167618  8750 raft_consensus.cc:504] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.168274  8750 leader_election.cc:288] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:13.168895  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65167136cc2d478cb0efd6a374839c48" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:13.169096  7724 raft_consensus.cc:2454] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 0.
I20240503 08:12:13.169483  7529 leader_election.cc:302] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:13.169780  8750 raft_consensus.cc:2781] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:13.169920  8750 raft_consensus.cc:482] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.170017  8750 raft_consensus.cc:3037] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.171995  8750 raft_consensus.cc:504] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.172466  8750 leader_election.cc:288] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:13.173156  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65167136cc2d478cb0efd6a374839c48" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:13.173346  7724 raft_consensus.cc:3037] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.175684  7724 raft_consensus.cc:2454] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 1.
I20240503 08:12:13.176126  7529 leader_election.cc:302] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:12:13.176584  8750 raft_consensus.cc:2781] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:13.176788  8750 raft_consensus.cc:686] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:13.177135  8750 consensus_queue.cc:237] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.180054  7447 catalog_manager.cc:5484] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } }
I20240503 08:12:13.221686  8575 raft_consensus.cc:482] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:13.221912  8575 raft_consensus.cc:504] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.222576  8575 leader_election.cc:288] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:13.223215  7853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ae2ad1ed4448689d8fe8eba7942dd9" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724" is_pre_election: true
I20240503 08:12:13.223423  7853 raft_consensus.cc:2454] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 0.
I20240503 08:12:13.223879  7658 leader_election.cc:302] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:13.224217  8575 raft_consensus.cc:2781] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:13.224336  8575 raft_consensus.cc:482] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.224464  8575 raft_consensus.cc:3037] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.226457  8575 raft_consensus.cc:504] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.226951  8575 leader_election.cc:288] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:13.227617  7853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ae2ad1ed4448689d8fe8eba7942dd9" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:12:13.227818  7853 raft_consensus.cc:3037] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.229848  7853 raft_consensus.cc:2454] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 1.
I20240503 08:12:13.230384  7658 leader_election.cc:302] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:13.230731  8575 raft_consensus.cc:2781] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:13.231189  8575 raft_consensus.cc:686] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Becoming Leader. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Running, Role: LEADER
I20240503 08:12:13.231483  8575 consensus_queue.cc:237] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } }
I20240503 08:12:13.234138  7447 catalog_manager.cc:5484] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: term changed from 0 to 1, leader changed from <none> to 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130). New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } }
I20240503 08:12:13.279275  8656 raft_consensus.cc:482] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:13.279482  8656 raft_consensus.cc:504] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.280063  8656 leader_election.cc:288] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:13.280622  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c94e75cff640f8927043b5e071c980" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b" is_pre_election: true
I20240503 08:12:13.280860  7724 raft_consensus.cc:2454] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 0.
I20240503 08:12:13.281291  7790 leader_election.cc:302] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:13.281566  8656 raft_consensus.cc:2781] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:13.281682  8656 raft_consensus.cc:482] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.281773  8656 raft_consensus.cc:3037] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.283749  8656 raft_consensus.cc:504] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.284217  8656 leader_election.cc:288] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:13.284838  7724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c94e75cff640f8927043b5e071c980" candidate_uuid: "5d1ee23344854913811909afd87ab724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:12:13.285058  7724 raft_consensus.cc:3037] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.286937  7724 raft_consensus.cc:2454] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1ee23344854913811909afd87ab724 in term 1.
I20240503 08:12:13.287418  7790 leader_election.cc:302] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:12:13.287739  8656 raft_consensus.cc:2781] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:13.288132  8656 raft_consensus.cc:686] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:12:13.288414  8656 consensus_queue.cc:237] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } }
I20240503 08:12:13.291006  7447 catalog_manager.cc:5484] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1ee23344854913811909afd87ab724 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } }
W20240503 08:12:13.662719  8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:13.663082  8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:13.670898  8777 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:12:13.754215  8750 consensus_queue.cc:1035] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.807997  8575 consensus_queue.cc:1035] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.864908  7724 raft_consensus.cc:1261] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:13.866271  8656 consensus_queue.cc:1035] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.920365  8817 mvcc.cc:205] Tried to move back new op lower bound from 7023509233103032320 to 7023509230751154176. Current Snapshot: MvccSnapshot[applied={T|T < 7023509233103032320}]

INSERT report
    rows total: 2000
    time total: 238.038 ms
  time per row: 0.119019 ms
W20240503 08:12:14.149622  7529 consensus_peers.cc:485] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 08:12:15.002663  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:41615
--local_ip_for_outbound_sockets=127.2.49.132
--tserver_master_addrs=127.2.49.190:41099
--webserver_port=40655
--webserver_interface=127.2.49.132
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:12:15.087131  7529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.132:41615: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:12:15.350531  8834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:15.350840  8834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:15.350979  8834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:15.358690  8834 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:15.358839  8834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:15.358999  8834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:12:15.369127  8834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:41615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=40655
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:15.370803  8834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:15.372630  8834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:15.385749  8844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:15.386370  8842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:15.386790  8841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:15.388049  8834 server_base.cc:1034] running on GCE node
I20240503 08:12:15.388783  8834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:15.389820  8834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:15.391031  8834 hybrid_clock.cc:648] HybridClock initialized: now 1714723935390964 us; error 59 us; skew 500 ppm
I20240503 08:12:15.391357  8834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:15.393951  8834 webserver.cc:460] Webserver started at http://127.2.49.132:40655/ using document root <none> and password file <none>
I20240503 08:12:15.394791  8834 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:15.394933  8834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:15.400738  8834 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240503 08:12:15.403930  8850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:15.405349  8834 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:12:15.405524  8834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb22180119ec428393402ddbd0b748b6"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:15.406507  8834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:15.432806  8834 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:15.433717  8834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:15.434029  8834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:15.435256  8834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:15.437420  8857 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:12:15.443233  8834 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:12:15.443377  8834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.001s
I20240503 08:12:15.443610  8834 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:12:15.446749  8834 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:12:15.446877  8834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:12:15.447245  8857 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Bootstrap starting.
I20240503 08:12:15.465924  8857 log.cc:824] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:15.469503  8857 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:15.470523  8857 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Bootstrap complete.
I20240503 08:12:15.472689  8857 ts_tablet_manager.cc:1367] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.007s
I20240503 08:12:15.488925  8964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:41615 every 8 connection(s)
I20240503 08:12:15.488891  8834 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:41615
I20240503 08:12:15.488864  8857 raft_consensus.cc:348] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:15.490253  8857 raft_consensus.cc:729] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Initialized, Role: FOLLOWER
I20240503 08:12:15.490975  8834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:12:15.491154  8857 consensus_queue.cc:260] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } }
I20240503 08:12:15.493310  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 8834
I20240503 08:12:15.493613  8857 ts_tablet_manager.cc:1398] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Time spent starting tablet: real 0.021s	user 0.017s	sys 0.004s
I20240503 08:12:15.494041  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:39615
--local_ip_for_outbound_sockets=127.2.49.133
--tserver_master_addrs=127.2.49.190:41099
--webserver_port=43589
--webserver_interface=127.2.49.133
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:12:15.498610  8967 tablet.cc:2369] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:15.505925  8965 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:15.506319  8965 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:15.507140  8965 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:15.510015  7444 ts_manager.cc:194] Re-registered known tserver with Master: fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615)
I20240503 08:12:15.512082  7444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:56787
I20240503 08:12:15.515034  8965 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
W20240503 08:12:15.848392  8970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:15.848718  8970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:15.848858  8970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:15.856946  8970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:15.857088  8970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:15.857245  8970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:12:15.866860  8970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:39615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=43589
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:15.868247  8970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:15.870188  8970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:15.883111  8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:15.884011  8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:15.884513  8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:15.885298  8970 server_base.cc:1034] running on GCE node
I20240503 08:12:15.886072  8970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:15.887558  8970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:15.888839  8970 hybrid_clock.cc:648] HybridClock initialized: now 1714723935888757 us; error 60 us; skew 500 ppm
I20240503 08:12:15.889268  8970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:15.892326  8970 webserver.cc:460] Webserver started at http://127.2.49.133:43589/ using document root <none> and password file <none>
I20240503 08:12:15.893353  8970 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:15.893517  8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:15.900998  8970 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:12:15.904345  8991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:15.906024  8970 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:12:15.906281  8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1456c568b524465bb505956695d13738"
format_stamp: "Formatted at 2024-05-03 08:12:02 on dist-test-slave-dz87"
I20240503 08:12:15.907218  8970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:15.928344  8970 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:15.929541  8970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:15.929953  8970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:15.931282  8970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:15.933199  8970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:15.933295  8970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:15.933447  8970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:15.933509  8970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:15.968016  8970 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:39615
I20240503 08:12:15.968058  9103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:39615 every 8 connection(s)
I20240503 08:12:15.969647  8970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:15.976997  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 8970
I20240503 08:12:15.977681  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.134:37149
--local_ip_for_outbound_sockets=127.2.49.134
--tserver_master_addrs=127.2.49.190:41099
--webserver_port=42933
--webserver_interface=127.2.49.134
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:15.984369  9104 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:15.984730  9104 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:15.985689  9104 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:15.987464  7444 ts_manager.cc:194] Re-registered known tserver with Master: 1456c568b524465bb505956695d13738 (127.2.49.133:39615)
I20240503 08:12:15.988301  7444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:36363
W20240503 08:12:16.335938  9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:16.336267  9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:16.336390  9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:16.344483  9107 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:16.344674  9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:16.344844  9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.134
I20240503 08:12:16.354516  9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.134:37149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.49.134
--webserver_port=42933
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.134
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:16.356099  9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:16.358018  9107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:16.371243  9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:16.371690  9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:16.372689  9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:16.373518  9107 server_base.cc:1034] running on GCE node
I20240503 08:12:16.374212  9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:16.375335  9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:16.376545  9107 hybrid_clock.cc:648] HybridClock initialized: now 1714723936376498 us; error 48 us; skew 500 ppm
I20240503 08:12:16.376878  9107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:16.379411  9107 webserver.cc:460] Webserver started at http://127.2.49.134:42933/ using document root <none> and password file <none>
I20240503 08:12:16.380234  9107 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:16.380383  9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:16.386355  9107 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:12:16.389621  9121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:16.391287  9107 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:12:16.391498  9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b50f0dfea2284eacbe4184b1f3ae3753"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:16.392416  9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:16.425838  9107 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:16.426865  9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:16.427186  9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:16.428268  9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:16.429905  9107 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:16.429996  9107 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:16.430131  9107 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:16.430223  9107 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240503 08:12:16.464020  9107 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.134:37149
I20240503 08:12:16.464075  9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.134:37149 every 8 connection(s)
I20240503 08:12:16.465586  9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:16.470996  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 9107
I20240503 08:12:16.471668  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.135:37243
--local_ip_for_outbound_sockets=127.2.49.135
--tserver_master_addrs=127.2.49.190:41099
--webserver_port=43435
--webserver_interface=127.2.49.135
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:12:16.480556  9234 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:16.480885  9234 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:16.481740  9234 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:16.483337  7444 ts_manager.cc:194] Re-registered known tserver with Master: b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149)
I20240503 08:12:16.484550  7444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.134:57945
W20240503 08:12:16.823154  9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:16.823478  9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:16.823606  9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:16.831444  9237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:12:16.831593  9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:16.831763  9237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.135
I20240503 08:12:16.841126  9237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.135:37243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.49.135
--webserver_port=43435
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.135
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:16.842506  9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:16.844218  9237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:16.857196  9245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:16.857812  9243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:16.858740  9242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:16.859710  9237 server_base.cc:1034] running on GCE node
I20240503 08:12:16.860383  9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:16.861558  9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:16.862818  9237 hybrid_clock.cc:648] HybridClock initialized: now 1714723936862752 us; error 56 us; skew 500 ppm
I20240503 08:12:16.863193  9237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:16.865574  9237 webserver.cc:460] Webserver started at http://127.2.49.135:43435/ using document root <none> and password file <none>
I20240503 08:12:16.866529  9237 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:16.866681  9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:16.872936  9237 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 08:12:16.876169  9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:16.877810  9237 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:12:16.878032  9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "131e5ac1bcce424cbb1f7f60be20a263"
format_stamp: "Formatted at 2024-05-03 08:12:03 on dist-test-slave-dz87"
I20240503 08:12:16.878989  9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:16.898770  9237 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:12:16.899648  9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:16.899950  9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:16.900923  9237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:16.902639  9237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:16.902736  9237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:16.902881  9237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:16.902951  9237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:12:16.958441  9237 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.135:37243
I20240503 08:12:16.958470  9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.135:37243 every 8 connection(s)
I20240503 08:12:16.959976  9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:16.964461  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 9237
I20240503 08:12:16.964790  2246 test_util.cc:268] Using random seed: 1193709712
I20240503 08:12:16.975243  9364 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:12:16.975582  9364 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:16.976385  9364 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:12:16.977802  7449 ts_manager.cc:194] Re-registered known tserver with Master: 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135:37243)
I20240503 08:12:16.978912  7449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.135:59569
I20240503 08:12:16.984027  2246 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 08:12:16.984582  2246 test_util.cc:268] Using random seed: 1193729500
I20240503 08:12:16.991681  9104 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
I20240503 08:12:17.001158  2246 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 08:12:17.002518  2246 test_util.cc:268] Using random seed: 1193747435
I20240503 08:12:17.037204  2246 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 08:12:17.038861  2246 test_util.cc:268] Using random seed: 1193783759
I20240503 08:12:17.068893  2246 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 08:12:17.137504  2246 test_util.cc:268] Using random seed: 1193882395
I20240503 08:12:17.211660  2246 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 08:12:17.487375  9234 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
I20240503 08:12:17.982385  9364 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
I20240503 08:12:18.756263  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:18.793581  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:18.840138  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:18.941287  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:18.965534  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:18.965664  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:18.991317  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:19.773113  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:19.864760  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:19.874694  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:19.881947  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:19.886641  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:19.895696  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:19.996181  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:20.335871  7591 consensus_queue.cc:237] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:20.342097  7854 raft_consensus.cc:1261] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 08:12:20.358055  8974 consensus_queue.cc:1035] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.009s
I20240503 08:12:20.369319  9445 raft_consensus.cc:2932] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:20.384246  7449 catalog_manager.cc:5484] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 reported cstate change: config changed from index -1 to 56, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
W20240503 08:12:20.385154  7527 consensus_peers.cc:485] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 -> Peer 1456c568b524465bb505956695d13738 (127.2.49.133:39615): Couldn't send request to peer 1456c568b524465bb505956695d13738. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f5a650ed26a406baa15105ac9fc8f8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:20.386449  7854 raft_consensus.cc:2932] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:20.803893  7852 consensus_queue.cc:237] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:20.824909  7721 raft_consensus.cc:1261] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 08:12:20.825871  9583 consensus_queue.cc:1035] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 08:12:20.829926  8656 raft_consensus.cc:2932] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:20.838430  7446 catalog_manager.cc:5484] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index -1 to 65, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:20.841413  7723 raft_consensus.cc:2932] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
W20240503 08:12:20.849326  7787 consensus_peers.cc:485] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 -> Peer b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149): Couldn't send request to peer b50f0dfea2284eacbe4184b1f3ae3753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e5a34b052e94b7b8a1dbaec2038d921. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:20.886310  9611 ts_tablet_manager.cc:902] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:20.912758  9611 tablet_copy_client.cc:308] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:12:21.104326  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet 9f5a650ed26a406baa15105ac9fc8f8f from peer 1456c568b524465bb505956695d13738 ({username='slave'} at 127.2.49.133:37013)
I20240503 08:12:21.104641  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet 9f5a650ed26a406baa15105ac9fc8f8f from peer 1456c568b524465bb505956695d13738 at {username='slave'} at 127.2.49.133:37013: session id = 1456c568b524465bb505956695d13738-9f5a650ed26a406baa15105ac9fc8f8f
I20240503 08:12:21.108125  7613 tablet_copy_source_session.cc:205] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:21.114794  9611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5a650ed26a406baa15105ac9fc8f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.150305  9611 tablet_copy_client.cc:783] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:21.150862  9611 tablet_copy_client.cc:647] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:21.155627  9611 tablet_copy_client.cc:523] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:21.169025  9611 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Bootstrap starting.
I20240503 08:12:21.231437  9611 log.cc:824] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:21.263440  7593 consensus_queue.cc:237] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:21.271747  7851 raft_consensus.cc:1261] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 08:12:21.277851  9461 consensus_queue.cc:1035] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 08:12:21.282140  8974 raft_consensus.cc:2932] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:21.289945  7852 raft_consensus.cc:2932] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:21.294066  7446 catalog_manager.cc:5484] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 reported cstate change: config changed from index -1 to 74, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:21.317924  9641 ts_tablet_manager.cc:902] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:21.380878  9611 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=696 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:21.381306  9641 tablet_copy_client.cc:308] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:12:21.381697  9611 tablet_bootstrap.cc:492] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Bootstrap complete.
I20240503 08:12:21.382427  9611 ts_tablet_manager.cc:1367] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Time spent bootstrapping tablet: real 0.214s	user 0.126s	sys 0.023s
W20240503 08:12:21.383522  7529 consensus_peers.cc:485] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 -> Peer 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135:37243): Couldn't send request to peer 131e5ac1bcce424cbb1f7f60be20a263. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c6b0211f6b44c1b92a5be884608d000. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:21.389873  9611 raft_consensus.cc:348] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:21.390861  9611 raft_consensus.cc:729] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456c568b524465bb505956695d13738, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.391901  9611 consensus_queue.cc:260] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:21.403908  9611 ts_tablet_manager.cc:1398] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738: Time spent starting tablet: real 0.021s	user 0.011s	sys 0.005s
I20240503 08:12:21.405841  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session 1456c568b524465bb505956695d13738-9f5a650ed26a406baa15105ac9fc8f8f received from {username='slave'} at 127.2.49.133:37013
I20240503 08:12:21.406064  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session 1456c568b524465bb505956695d13738-9f5a650ed26a406baa15105ac9fc8f8f on tablet 9f5a650ed26a406baa15105ac9fc8f8f with peer 1456c568b524465bb505956695d13738
I20240503 08:12:21.438467  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 7e5a34b052e94b7b8a1dbaec2038d921 from peer b50f0dfea2284eacbe4184b1f3ae3753 ({username='slave'} at 127.2.49.134:41109)
I20240503 08:12:21.438771  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 7e5a34b052e94b7b8a1dbaec2038d921 from peer b50f0dfea2284eacbe4184b1f3ae3753 at {username='slave'} at 127.2.49.134:41109: session id = b50f0dfea2284eacbe4184b1f3ae3753-7e5a34b052e94b7b8a1dbaec2038d921
I20240503 08:12:21.443248  7874 tablet_copy_source_session.cc:205] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:21.449630  9641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5a34b052e94b7b8a1dbaec2038d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:21.467701  9641 tablet_copy_client.cc:783] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:21.468205  9641 tablet_copy_client.cc:647] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:21.471446  9641 tablet_copy_client.cc:523] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:21.480383  9641 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap starting.
I20240503 08:12:21.514052  9059 raft_consensus.cc:1203] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.71]   Dedup: 1.65->[1.66-1.71]
I20240503 08:12:21.522325  9641 log.cc:824] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:21.787164  9641 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:21.787892  9641 tablet_bootstrap.cc:492] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap complete.
I20240503 08:12:21.788532  9641 ts_tablet_manager.cc:1367] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Time spent bootstrapping tablet: real 0.311s	user 0.151s	sys 0.023s
I20240503 08:12:21.796478  9641 raft_consensus.cc:348] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:21.797125  9641 raft_consensus.cc:729] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50f0dfea2284eacbe4184b1f3ae3753, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.799547  9641 consensus_queue.cc:260] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:21.805564  9641 ts_tablet_manager.cc:1398] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753: Time spent starting tablet: real 0.017s	user 0.011s	sys 0.001s
I20240503 08:12:21.807865  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-7e5a34b052e94b7b8a1dbaec2038d921 received from {username='slave'} at 127.2.49.134:41109
I20240503 08:12:21.808060  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-7e5a34b052e94b7b8a1dbaec2038d921 on tablet 7e5a34b052e94b7b8a1dbaec2038d921 with peer b50f0dfea2284eacbe4184b1f3ae3753
I20240503 08:12:21.830600  9667 ts_tablet_manager.cc:902] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:21.930092  9189 raft_consensus.cc:1203] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.81]   Dedup: 1.75->[1.76-1.81]
I20240503 08:12:22.027127  9667 tablet_copy_client.cc:308] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:12:22.060926  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet 6c6b0211f6b44c1b92a5be884608d000 from peer 131e5ac1bcce424cbb1f7f60be20a263 ({username='slave'} at 127.2.49.135:54619)
I20240503 08:12:22.061131  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet 6c6b0211f6b44c1b92a5be884608d000 from peer 131e5ac1bcce424cbb1f7f60be20a263 at {username='slave'} at 127.2.49.135:54619: session id = 131e5ac1bcce424cbb1f7f60be20a263-6c6b0211f6b44c1b92a5be884608d000
I20240503 08:12:22.064249  7613 tablet_copy_source_session.cc:205] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:22.070685  9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c6b0211f6b44c1b92a5be884608d000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.094691  9667 tablet_copy_client.cc:783] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:22.095173  9667 tablet_copy_client.cc:647] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:22.097927  9667 tablet_copy_client.cc:523] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:22.104238  9667 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap starting.
I20240503 08:12:22.140628  9667 log.cc:824] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:22.207481  7593 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:22.209733  7722 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 08:12:22.210435  9638 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
W20240503 08:12:22.211475  7529 consensus_peers.cc:485] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80eccc47e060493ebe43b007c7a23f00. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:22.214334  9638 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:22.216037  7722 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:22.223397  7449 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index -1 to 70, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:22.338765  9667 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=85 ignored=0} inserts{seen=770 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:22.339637  9667 tablet_bootstrap.cc:492] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap complete.
I20240503 08:12:22.340498  9667 ts_tablet_manager.cc:1367] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Time spent bootstrapping tablet: real 0.237s	user 0.166s	sys 0.012s
I20240503 08:12:22.349483  9667 raft_consensus.cc:348] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:22.360306  9667 raft_consensus.cc:729] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131e5ac1bcce424cbb1f7f60be20a263, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.361374  9667 consensus_queue.cc:260] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:22.373049  9667 ts_tablet_manager.cc:1398] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263: Time spent starting tablet: real 0.032s	user 0.020s	sys 0.006s
I20240503 08:12:22.377921  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-6c6b0211f6b44c1b92a5be884608d000 received from {username='slave'} at 127.2.49.135:54619
I20240503 08:12:22.378070  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-6c6b0211f6b44c1b92a5be884608d000 on tablet 6c6b0211f6b44c1b92a5be884608d000 with peer 131e5ac1bcce424cbb1f7f60be20a263
I20240503 08:12:22.725791  7854 consensus_queue.cc:237] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:22.731786  7720 raft_consensus.cc:1261] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 08:12:22.736647  9419 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 08:12:22.742301  9699 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index -1 to 107, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:22.756207  9713 ts_tablet_manager.cc:902] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:22.756129  7446 catalog_manager.cc:5484] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index -1 to 107, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:22.756724  7720 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index -1 to 107, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:22.767469  9319 raft_consensus.cc:1203] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.98]   Dedup: 1.86->[1.87-1.98]
W20240503 08:12:22.803189  7788 consensus_peers.cc:485] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 -> Peer 1456c568b524465bb505956695d13738 (127.2.49.133:39615): Couldn't send request to peer 1456c568b524465bb505956695d13738. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6462c70bafc94b7fb3bcb9502e80e096. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:22.916524  9713 tablet_copy_client.cc:308] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:12:23.088650  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet 80eccc47e060493ebe43b007c7a23f00 from peer fb22180119ec428393402ddbd0b748b6 ({username='slave'} at 127.2.49.132:37923)
I20240503 08:12:23.088845  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet 80eccc47e060493ebe43b007c7a23f00 from peer fb22180119ec428393402ddbd0b748b6 at {username='slave'} at 127.2.49.132:37923: session id = fb22180119ec428393402ddbd0b748b6-80eccc47e060493ebe43b007c7a23f00
I20240503 08:12:23.095547  7613 tablet_copy_source_session.cc:205] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:23.106542  9713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eccc47e060493ebe43b007c7a23f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.117159  9713 tablet_copy_client.cc:783] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:23.117758  9713 tablet_copy_client.cc:647] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:23.138913  9713 tablet_copy_client.cc:523] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:23.143185  9713 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: Bootstrap starting.
I20240503 08:12:23.205334  7852 consensus_queue.cc:237] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
W20240503 08:12:23.208956  7787 consensus_peers.cc:485] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 -> Peer b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149): Couldn't send request to peer b50f0dfea2284eacbe4184b1f3ae3753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8967d6dac1aa482683c8bad1892dcbde. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:23.216446  7592 raft_consensus.cc:1261] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 08:12:23.217823  9419 consensus_queue.cc:1035] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240503 08:12:23.252218  9419 raft_consensus.cc:2932] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:23.257078  7593 raft_consensus.cc:2932] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:23.263558  7446 catalog_manager.cc:5484] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index -1 to 95, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:23.408391  9713 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:23.410646  9713 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: Bootstrap complete.
I20240503 08:12:23.411113  9713 ts_tablet_manager.cc:1367] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: Time spent bootstrapping tablet: real 0.268s	user 0.141s	sys 0.013s
I20240503 08:12:23.415978  9713 raft_consensus.cc:348] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:23.419771  9713 raft_consensus.cc:729] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.420264  9713 consensus_queue.cc:260] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:23.574803  9745 ts_tablet_manager.cc:902] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:23.632611  9713 ts_tablet_manager.cc:1398] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6: Time spent starting tablet: real 0.221s	user 0.050s	sys 0.055s
I20240503 08:12:23.644042  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session fb22180119ec428393402ddbd0b748b6-80eccc47e060493ebe43b007c7a23f00 received from {username='slave'} at 127.2.49.132:37923
I20240503 08:12:23.644254  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session fb22180119ec428393402ddbd0b748b6-80eccc47e060493ebe43b007c7a23f00 on tablet 80eccc47e060493ebe43b007c7a23f00 with peer fb22180119ec428393402ddbd0b748b6
I20240503 08:12:23.703670  9745 tablet_copy_client.cc:308] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:12:23.709661  9765 ts_tablet_manager.cc:902] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:23.733150  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 6462c70bafc94b7fb3bcb9502e80e096 from peer 1456c568b524465bb505956695d13738 ({username='slave'} at 127.2.49.133:60209)
I20240503 08:12:23.733361  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 6462c70bafc94b7fb3bcb9502e80e096 from peer 1456c568b524465bb505956695d13738 at {username='slave'} at 127.2.49.133:60209: session id = 1456c568b524465bb505956695d13738-6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:12:23.736110  7874 tablet_copy_source_session.cc:205] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:23.740006  9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6462c70bafc94b7fb3bcb9502e80e096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.749397  9745 tablet_copy_client.cc:783] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:23.749935  9745 tablet_copy_client.cc:647] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:23.752045  9765 tablet_copy_client.cc:308] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:12:23.753425  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 8967d6dac1aa482683c8bad1892dcbde from peer b50f0dfea2284eacbe4184b1f3ae3753 ({username='slave'} at 127.2.49.134:41109)
I20240503 08:12:23.753561  9745 tablet_copy_client.cc:523] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:23.753757  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 8967d6dac1aa482683c8bad1892dcbde from peer b50f0dfea2284eacbe4184b1f3ae3753 at {username='slave'} at 127.2.49.134:41109: session id = b50f0dfea2284eacbe4184b1f3ae3753-8967d6dac1aa482683c8bad1892dcbde
I20240503 08:12:23.759888  7874 tablet_copy_source_session.cc:205] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:23.760103  9745 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: Bootstrap starting.
I20240503 08:12:23.761793  9765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8967d6dac1aa482683c8bad1892dcbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.771615  9765 tablet_copy_client.cc:783] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:23.772120  9765 tablet_copy_client.cc:647] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:23.775049  9765 tablet_copy_client.cc:523] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:23.784035  9765 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap starting.
I20240503 08:12:23.832726  7851 consensus_queue.cc:237] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:23.850297  7593 raft_consensus.cc:1261] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 08:12:23.855532  9394 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
W20240503 08:12:23.882334  7790 consensus_peers.cc:485] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 -> Peer 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135:37243): Couldn't send request to peer 131e5ac1bcce424cbb1f7f60be20a263. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 303fded976c342efbccfc375950f78ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:23.902037  7593 raft_consensus.cc:1203] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.101->[1.102-1.103]   Dedup: 1.102->[1.103-1.103]
I20240503 08:12:23.911401  9394 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index -1 to 103, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:23.920796  7593 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:23.957810  7447 catalog_manager.cc:5484] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 reported cstate change: config changed from index -1 to 103, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:23.959937  8920 raft_consensus.cc:1203] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.90]   Dedup: 1.84->[1.85-1.90]
I20240503 08:12:24.065246  9765 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=98 ignored=0} inserts{seen=729 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:24.066131  9765 tablet_bootstrap.cc:492] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap complete.
I20240503 08:12:24.066605  9765 ts_tablet_manager.cc:1367] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: Time spent bootstrapping tablet: real 0.283s	user 0.157s	sys 0.028s
I20240503 08:12:24.067747  9765 raft_consensus.cc:348] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:24.068078  9765 raft_consensus.cc:729] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50f0dfea2284eacbe4184b1f3ae3753, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.068403  9765 consensus_queue.cc:260] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:24.076174  9765 ts_tablet_manager.cc:1398] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.007s
I20240503 08:12:24.079345  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-8967d6dac1aa482683c8bad1892dcbde received from {username='slave'} at 127.2.49.134:41109
I20240503 08:12:24.079552  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-8967d6dac1aa482683c8bad1892dcbde on tablet 8967d6dac1aa482683c8bad1892dcbde with peer b50f0dfea2284eacbe4184b1f3ae3753
I20240503 08:12:24.133709  9745 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: Bootstrap replayed 1/1 log segments. Stats: ops{read=113 overwritten=0 applied=113 ignored=0} inserts{seen=798 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:24.134346  9745 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: Bootstrap complete.
I20240503 08:12:24.134682  9745 ts_tablet_manager.cc:1367] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: Time spent bootstrapping tablet: real 0.375s	user 0.161s	sys 0.021s
I20240503 08:12:24.135782  9745 raft_consensus.cc:348] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:24.136550  9745 raft_consensus.cc:729] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456c568b524465bb505956695d13738, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.136910  9745 consensus_queue.cc:260] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 113, Last appended: 1.113, Last appended by leader: 113, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:24.144369  9745 ts_tablet_manager.cc:1398] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.006s
I20240503 08:12:24.149278  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session 1456c568b524465bb505956695d13738-6462c70bafc94b7fb3bcb9502e80e096 received from {username='slave'} at 127.2.49.133:60209
I20240503 08:12:24.149482  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session 1456c568b524465bb505956695d13738-6462c70bafc94b7fb3bcb9502e80e096 on tablet 6462c70bafc94b7fb3bcb9502e80e096 with peer 1456c568b524465bb505956695d13738
I20240503 08:12:24.209084  9189 raft_consensus.cc:1203] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.94->[1.95-1.99]   Dedup: 1.98->[1.99-1.99]
I20240503 08:12:24.260229  9058 raft_consensus.cc:1203] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.106->[1.107-1.119]   Dedup: 1.113->[1.114-1.119]
I20240503 08:12:24.331399  9806 ts_tablet_manager.cc:902] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:24.479343  9806 tablet_copy_client.cc:308] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:12:24.558091  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 303fded976c342efbccfc375950f78ec from peer 131e5ac1bcce424cbb1f7f60be20a263 ({username='slave'} at 127.2.49.135:50457)
I20240503 08:12:24.558336  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 303fded976c342efbccfc375950f78ec from peer 131e5ac1bcce424cbb1f7f60be20a263 at {username='slave'} at 127.2.49.135:50457: session id = 131e5ac1bcce424cbb1f7f60be20a263-303fded976c342efbccfc375950f78ec
I20240503 08:12:24.561123  7874 tablet_copy_source_session.cc:205] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:24.568169  9806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303fded976c342efbccfc375950f78ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.575349  9806 tablet_copy_client.cc:783] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:24.575711  9806 tablet_copy_client.cc:647] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:24.586277  9806 tablet_copy_client.cc:523] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:24.590307  9806 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap starting.
I20240503 08:12:24.625034  7593 consensus_queue.cc:237] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:24.627753  7724 raft_consensus.cc:1261] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240503 08:12:24.628692  9561 consensus_queue.cc:1035] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
W20240503 08:12:24.631631  7529 consensus_peers.cc:485] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65167136cc2d478cb0efd6a374839c48. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:24.638423  8974 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:24.645789  7719 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:24.651746  7449 catalog_manager.cc:5484] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index -1 to 113, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:24.819464  9806 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:24.820029  9806 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap complete.
I20240503 08:12:24.820386  9806 ts_tablet_manager.cc:1367] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: Time spent bootstrapping tablet: real 0.230s	user 0.161s	sys 0.013s
I20240503 08:12:24.821472  9806 raft_consensus.cc:348] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:24.821820  9806 raft_consensus.cc:729] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131e5ac1bcce424cbb1f7f60be20a263, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.822116  9806 consensus_queue.cc:260] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:24.829425  9806 ts_tablet_manager.cc:1398] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240503 08:12:24.830322  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-303fded976c342efbccfc375950f78ec received from {username='slave'} at 127.2.49.135:50457
I20240503 08:12:24.830462  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-303fded976c342efbccfc375950f78ec on tablet 303fded976c342efbccfc375950f78ec with peer 131e5ac1bcce424cbb1f7f60be20a263
I20240503 08:12:24.882676  9319 raft_consensus.cc:1203] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.102->[1.103-1.112]   Dedup: 1.108->[1.109-1.112]
I20240503 08:12:25.102172  7592 consensus_queue.cc:237] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
W20240503 08:12:25.123983  7527 consensus_peers.cc:485] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 -> Peer 1456c568b524465bb505956695d13738 (127.2.49.133:39615): Couldn't send request to peer 1456c568b524465bb505956695d13738. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0246c726f68848a7ad4620e533f93b53. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:25.126235  8920 raft_consensus.cc:1261] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:25.138346  9762 consensus_queue.cc:1035] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 08:12:25.169349  9762 raft_consensus.cc:2932] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:25.170770  8920 raft_consensus.cc:2932] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:25.180397  7449 catalog_manager.cc:5484] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 reported cstate change: config changed from index -1 to 2, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:25.223520  9854 ts_tablet_manager.cc:902] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:25.358146  9854 tablet_copy_client.cc:308] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:12:25.359440  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet 65167136cc2d478cb0efd6a374839c48 from peer fb22180119ec428393402ddbd0b748b6 ({username='slave'} at 127.2.49.132:37923)
I20240503 08:12:25.359642  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet 65167136cc2d478cb0efd6a374839c48 from peer fb22180119ec428393402ddbd0b748b6 at {username='slave'} at 127.2.49.132:37923: session id = fb22180119ec428393402ddbd0b748b6-65167136cc2d478cb0efd6a374839c48
I20240503 08:12:25.388232  7613 tablet_copy_source_session.cc:205] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:25.389873  9854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65167136cc2d478cb0efd6a374839c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:25.396471  9854 tablet_copy_client.cc:783] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:25.396828  9854 tablet_copy_client.cc:647] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:25.421373  9854 tablet_copy_client.cc:523] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:25.426064  9854 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Bootstrap starting.
I20240503 08:12:25.592638  9878 ts_tablet_manager.cc:902] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:25.614295  7724 consensus_queue.cc:237] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 1.144, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:25.627064  9878 tablet_copy_client.cc:308] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:12:25.628047  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet 0246c726f68848a7ad4620e533f93b53 from peer 1456c568b524465bb505956695d13738 ({username='slave'} at 127.2.49.133:37013)
I20240503 08:12:25.628285  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet 0246c726f68848a7ad4620e533f93b53 from peer 1456c568b524465bb505956695d13738 at {username='slave'} at 127.2.49.133:37013: session id = 1456c568b524465bb505956695d13738-0246c726f68848a7ad4620e533f93b53
I20240503 08:12:25.634052  7613 tablet_copy_source_session.cc:205] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:25.637390  9878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0246c726f68848a7ad4620e533f93b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:25.649042  7854 raft_consensus.cc:1261] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch)
I20240503 08:12:25.654068  9878 tablet_copy_client.cc:783] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:25.654448  9878 tablet_copy_client.cc:647] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:25.657465  9878 tablet_copy_client.cc:523] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:25.661180  9878 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: Bootstrap starting.
I20240503 08:12:25.666813  9633 consensus_queue.cc:1035] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 144, Time since last communication: 0.000s
I20240503 08:12:25.681522  9878 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:25.682147  9878 tablet_bootstrap.cc:492] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: Bootstrap complete.
I20240503 08:12:25.682512  9878 ts_tablet_manager.cc:1367] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.010s
I20240503 08:12:25.683534  9878 raft_consensus.cc:348] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:25.683881  9878 raft_consensus.cc:729] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456c568b524465bb505956695d13738, State: Initialized, Role: FOLLOWER
I20240503 08:12:25.684254  9878 consensus_queue.cc:260] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:25.692319  9878 ts_tablet_manager.cc:1398] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20240503 08:12:25.698323  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session 1456c568b524465bb505956695d13738-0246c726f68848a7ad4620e533f93b53 received from {username='slave'} at 127.2.49.133:37013
I20240503 08:12:25.698505  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session 1456c568b524465bb505956695d13738-0246c726f68848a7ad4620e533f93b53 on tablet 0246c726f68848a7ad4620e533f93b53 with peer 1456c568b524465bb505956695d13738
I20240503 08:12:25.698249  9634 raft_consensus.cc:2932] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.145: config changed from index -1 to 145, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:25.705756  7449 catalog_manager.cc:5484] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index -1 to 145, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:25.714375  7854 raft_consensus.cc:2932] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index -1 to 145, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:25.766505  9854 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=124 overwritten=0 applied=124 ignored=0} inserts{seen=796 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:25.767103  9854 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Bootstrap complete.
I20240503 08:12:25.767488  9854 ts_tablet_manager.cc:1367] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Time spent bootstrapping tablet: real 0.342s	user 0.201s	sys 0.035s
I20240503 08:12:25.768579  9854 raft_consensus.cc:348] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:25.768914  9854 raft_consensus.cc:729] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Initialized, Role: FOLLOWER
I20240503 08:12:25.769241  9854 consensus_queue.cc:260] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 124, Last appended: 1.124, Last appended by leader: 124, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:25.778909  9854 ts_tablet_manager.cc:1398] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20240503 08:12:25.785455  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session fb22180119ec428393402ddbd0b748b6-65167136cc2d478cb0efd6a374839c48 received from {username='slave'} at 127.2.49.132:37923
I20240503 08:12:25.785641  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session fb22180119ec428393402ddbd0b748b6-65167136cc2d478cb0efd6a374839c48 on tablet 65167136cc2d478cb0efd6a374839c48 with peer fb22180119ec428393402ddbd0b748b6
W20240503 08:12:25.794704  7660 consensus_peers.cc:485] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b -> Peer 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135:37243): Couldn't send request to peer 131e5ac1bcce424cbb1f7f60be20a263. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91ae2ad1ed4448689d8fe8eba7942dd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:26.073890  8919 raft_consensus.cc:1203] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.112->[1.113-1.133]   Dedup: 1.124->[1.125-1.133]
I20240503 08:12:26.106976  9059 raft_consensus.cc:1203] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240503 08:12:26.148094  7854 consensus_queue.cc:237] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.127, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
W20240503 08:12:26.152920  7787 consensus_peers.cc:485] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 -> Peer b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149): Couldn't send request to peer b50f0dfea2284eacbe4184b1f3ae3753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44c94e75cff640f8927043b5e071c980. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:26.153332  7719 raft_consensus.cc:1261] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 08:12:26.153859  9873 consensus_queue.cc:1035] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20240503 08:12:26.156697  9394 raft_consensus.cc:2932] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.128: config changed from index -1 to 128, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:26.158524  7719 raft_consensus.cc:2932] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index -1 to 128, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:26.163203  7445 catalog_manager.cc:5484] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index -1 to 128, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:26.247972  9916 ts_tablet_manager.cc:902] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: Initiating tablet copy from peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:26.306001  9916 tablet_copy_client.cc:308] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:43861
I20240503 08:12:26.352005  7744 tablet_copy_service.cc:140] P 359f2f323e79431eac07dc0a3f47a47b: Received BeginTabletCopySession request for tablet 91ae2ad1ed4448689d8fe8eba7942dd9 from peer 131e5ac1bcce424cbb1f7f60be20a263 ({username='slave'} at 127.2.49.135:60441)
I20240503 08:12:26.352337  7744 tablet_copy_service.cc:161] P 359f2f323e79431eac07dc0a3f47a47b: Beginning new tablet copy session on tablet 91ae2ad1ed4448689d8fe8eba7942dd9 from peer 131e5ac1bcce424cbb1f7f60be20a263 at {username='slave'} at 127.2.49.135:60441: session id = 131e5ac1bcce424cbb1f7f60be20a263-91ae2ad1ed4448689d8fe8eba7942dd9
I20240503 08:12:26.359249  7744 tablet_copy_source_session.cc:205] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:26.368767  9916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ae2ad1ed4448689d8fe8eba7942dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.376250  9916 tablet_copy_client.cc:783] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:26.376646  9916 tablet_copy_client.cc:647] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:26.398862  9916 tablet_copy_client.cc:523] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:26.403754  9916 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap starting.
I20240503 08:12:26.471060  7724 consensus_queue.cc:237] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 152, Committed index: 152, Last appended: 1.153, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:26.514441  7851 raft_consensus.cc:1261] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
W20240503 08:12:26.514950  7660 consensus_peers.cc:485] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01f87cf2ca7041cd8727e41d994f330c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:26.523415  9433 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 152, Time since last communication: 0.000s
I20240503 08:12:26.552193  9433 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.154: config changed from index -1 to 154, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:26.567754  7854 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index -1 to 154, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:26.569412  7445 catalog_manager.cc:5484] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index -1 to 154, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:26.581331  9945 ts_tablet_manager.cc:902] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:26.765305  9945 tablet_copy_client.cc:308] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:12:26.768824  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 44c94e75cff640f8927043b5e071c980 from peer b50f0dfea2284eacbe4184b1f3ae3753 ({username='slave'} at 127.2.49.134:41109)
I20240503 08:12:26.769058  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 44c94e75cff640f8927043b5e071c980 from peer b50f0dfea2284eacbe4184b1f3ae3753 at {username='slave'} at 127.2.49.134:41109: session id = b50f0dfea2284eacbe4184b1f3ae3753-44c94e75cff640f8927043b5e071c980
I20240503 08:12:26.771903  7874 tablet_copy_source_session.cc:205] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:26.775409  9945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c94e75cff640f8927043b5e071c980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.782871  9945 tablet_copy_client.cc:783] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:26.783231  9945 tablet_copy_client.cc:647] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:26.849575  9945 tablet_copy_client.cc:523] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:26.853240  9945 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap starting.
I20240503 08:12:27.009510  9960 ts_tablet_manager.cc:902] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: Initiating tablet copy from peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:12:27.115303  9916 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap replayed 1/1 log segments. Stats: ops{read=152 overwritten=0 applied=151 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:27.118826  9916 tablet_bootstrap.cc:492] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap complete.
I20240503 08:12:27.122978  9916 ts_tablet_manager.cc:1367] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: Time spent bootstrapping tablet: real 0.719s	user 0.237s	sys 0.051s
I20240503 08:12:27.131021  9916 raft_consensus.cc:348] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:27.133139  9916 raft_consensus.cc:729] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131e5ac1bcce424cbb1f7f60be20a263, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.133821  9916 consensus_queue.cc:260] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 1.152, Last appended by leader: 152, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:27.148607  9916 ts_tablet_manager.cc:1398] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20240503 08:12:27.153260  7744 tablet_copy_service.cc:342] P 359f2f323e79431eac07dc0a3f47a47b: Request end of tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-91ae2ad1ed4448689d8fe8eba7942dd9 received from {username='slave'} at 127.2.49.135:60441
I20240503 08:12:27.153483  7744 tablet_copy_service.cc:433] P 359f2f323e79431eac07dc0a3f47a47b: ending tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-91ae2ad1ed4448689d8fe8eba7942dd9 on tablet 91ae2ad1ed4448689d8fe8eba7942dd9 with peer 131e5ac1bcce424cbb1f7f60be20a263
W20240503 08:12:27.386086  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:43861, user_credentials={real_user=slave}} blocked reactor thread for 46604.1us
I20240503 08:12:27.392498  9319 raft_consensus.cc:1203] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.144->[1.145-1.168]   Dedup: 1.152->[1.153-1.168]
W20240503 08:12:27.541945  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:43861, user_credentials={real_user=slave}} blocked reactor thread for 69423us
I20240503 08:12:27.612283  9945 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap replayed 1/1 log segments. Stats: ops{read=135 overwritten=0 applied=135 ignored=0} inserts{seen=820 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:27.612811  9945 tablet_bootstrap.cc:492] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap complete.
I20240503 08:12:27.613265  9945 ts_tablet_manager.cc:1367] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: Time spent bootstrapping tablet: real 0.760s	user 0.212s	sys 0.010s
I20240503 08:12:27.617069  9945 raft_consensus.cc:348] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:27.617391  9945 raft_consensus.cc:729] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50f0dfea2284eacbe4184b1f3ae3753, State: Initialized, Role: FOLLOWER
I20240503 08:12:27.617671  9945 consensus_queue.cc:260] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 135, Last appended: 1.135, Last appended by leader: 135, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
W20240503 08:12:27.628089  9960 thread.cc:636] tablet-download-pool-01f87cf2ca7041cd8727e41d994f330c [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.109s	sys 0.156s
W20240503 08:12:27.628337  9960 thread.cc:602] tablet-download-pool-01f87cf2ca7041cd8727e41d994f330c [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.109s	sys 0.156s
I20240503 08:12:27.629029  9960 tablet_copy_client.cc:308] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:43861
I20240503 08:12:27.654290  7744 tablet_copy_service.cc:140] P 359f2f323e79431eac07dc0a3f47a47b: Received BeginTabletCopySession request for tablet 01f87cf2ca7041cd8727e41d994f330c from peer fb22180119ec428393402ddbd0b748b6 ({username='slave'} at 127.2.49.132:49375)
I20240503 08:12:27.654489  7744 tablet_copy_service.cc:161] P 359f2f323e79431eac07dc0a3f47a47b: Beginning new tablet copy session on tablet 01f87cf2ca7041cd8727e41d994f330c from peer fb22180119ec428393402ddbd0b748b6 at {username='slave'} at 127.2.49.132:49375: session id = fb22180119ec428393402ddbd0b748b6-01f87cf2ca7041cd8727e41d994f330c
I20240503 08:12:27.657707  7744 tablet_copy_source_session.cc:205] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:27.659160  9960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f87cf2ca7041cd8727e41d994f330c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:27.659634  9945 ts_tablet_manager.cc:1398] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753: Time spent starting tablet: real 0.046s	user 0.029s	sys 0.004s
I20240503 08:12:27.661880  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-44c94e75cff640f8927043b5e071c980 received from {username='slave'} at 127.2.49.134:41109
I20240503 08:12:27.662063  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-44c94e75cff640f8927043b5e071c980 on tablet 44c94e75cff640f8927043b5e071c980 with peer b50f0dfea2284eacbe4184b1f3ae3753
I20240503 08:12:27.671546  9960 tablet_copy_client.cc:783] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:27.671963  9960 tablet_copy_client.cc:647] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:27.676466  9960 tablet_copy_client.cc:523] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:27.680063  9960 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: Bootstrap starting.
I20240503 08:12:27.707481  7854 consensus_queue.cc:237] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 1.166, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
W20240503 08:12:27.715020  7790 consensus_peers.cc:485] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 -> Peer 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135:37243): Couldn't send request to peer 131e5ac1bcce424cbb1f7f60be20a263. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a629b67231f84cdbb5e302265896e25a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:27.718930  7724 raft_consensus.cc:1261] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20240503 08:12:27.725273  9809 consensus_queue.cc:1035] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 08:12:27.736660  9394 raft_consensus.cc:2932] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.167: config changed from index -1 to 167, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:27.742529  9187 raft_consensus.cc:1203] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.127->[1.128-1.150]   Dedup: 1.135->[1.136-1.150]
I20240503 08:12:27.754411  7445 catalog_manager.cc:5484] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index -1 to 167, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:27.757462  7720 raft_consensus.cc:2932] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index -1 to 167, VOTER 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135) added. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:28.247397 10007 ts_tablet_manager.cc:902] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:12:28.427850  9960 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=167 overwritten=0 applied=167 ignored=0} inserts{seen=798 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:28.428455  9960 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: Bootstrap complete.
I20240503 08:12:28.428843  9960 ts_tablet_manager.cc:1367] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: Time spent bootstrapping tablet: real 0.749s	user 0.251s	sys 0.030s
I20240503 08:12:28.429750  9960 raft_consensus.cc:348] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:28.430117  9960 raft_consensus.cc:729] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Initialized, Role: FOLLOWER
I20240503 08:12:28.430465  9960 consensus_queue.cc:260] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 167, Last appended: 1.167, Last appended by leader: 167, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:28.438055  9960 ts_tablet_manager.cc:1398] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240503 08:12:28.443225  7744 tablet_copy_service.cc:342] P 359f2f323e79431eac07dc0a3f47a47b: Request end of tablet copy session fb22180119ec428393402ddbd0b748b6-01f87cf2ca7041cd8727e41d994f330c received from {username='slave'} at 127.2.49.132:49375
I20240503 08:12:28.443424  7744 tablet_copy_service.cc:433] P 359f2f323e79431eac07dc0a3f47a47b: ending tablet copy session fb22180119ec428393402ddbd0b748b6-01f87cf2ca7041cd8727e41d994f330c on tablet 01f87cf2ca7041cd8727e41d994f330c with peer fb22180119ec428393402ddbd0b748b6
I20240503 08:12:28.563830  8919 raft_consensus.cc:1203] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.153->[1.154-1.174]   Dedup: 1.167->[1.168-1.174]
I20240503 08:12:28.686376 10007 tablet_copy_client.cc:308] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:12:28.687834  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet a629b67231f84cdbb5e302265896e25a from peer 131e5ac1bcce424cbb1f7f60be20a263 ({username='slave'} at 127.2.49.135:50457)
I20240503 08:12:28.688047  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet a629b67231f84cdbb5e302265896e25a from peer 131e5ac1bcce424cbb1f7f60be20a263 at {username='slave'} at 127.2.49.135:50457: session id = 131e5ac1bcce424cbb1f7f60be20a263-a629b67231f84cdbb5e302265896e25a
I20240503 08:12:28.691022  7874 tablet_copy_source_session.cc:205] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:28.692528 10007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a629b67231f84cdbb5e302265896e25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:28.702224 10007 tablet_copy_client.cc:783] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:28.702575 10007 tablet_copy_client.cc:647] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:28.705386 10007 tablet_copy_client.cc:523] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:28.709558 10007 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap starting.
W20240503 08:12:28.722476  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 94965us
I20240503 08:12:29.239234 10007 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap replayed 1/1 log segments. Stats: ops{read=176 overwritten=0 applied=176 ignored=0} inserts{seen=807 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:29.239885 10007 tablet_bootstrap.cc:492] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: Bootstrap complete.
I20240503 08:12:29.241875 10007 ts_tablet_manager.cc:1367] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: Time spent bootstrapping tablet: real 0.532s	user 0.262s	sys 0.026s
I20240503 08:12:29.250269 10007 raft_consensus.cc:348] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:29.251673 10007 raft_consensus.cc:729] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131e5ac1bcce424cbb1f7f60be20a263, State: Initialized, Role: FOLLOWER
I20240503 08:12:29.252808 10007 consensus_queue.cc:260] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 176, Last appended: 1.176, Last appended by leader: 176, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:29.274050 10007 ts_tablet_manager.cc:1398] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20240503 08:12:29.283064  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-a629b67231f84cdbb5e302265896e25a received from {username='slave'} at 127.2.49.135:50457
I20240503 08:12:29.283237  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session 131e5ac1bcce424cbb1f7f60be20a263-a629b67231f84cdbb5e302265896e25a on tablet a629b67231f84cdbb5e302265896e25a with peer 131e5ac1bcce424cbb1f7f60be20a263
I20240503 08:12:29.507632  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:29.603317  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:29.741138  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:29.896620  9317 raft_consensus.cc:1203] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.166->[1.167-1.186]   Dedup: 1.176->[1.177-1.186]
I20240503 08:12:30.093627  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:30.139835  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:30.160445  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:30.327677  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:30.861655  7723 consensus_queue.cc:237] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:30.879927  9318 raft_consensus.cc:1261] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20240503 08:12:30.895792  9952 consensus_queue.cc:1035] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240503 08:12:30.917930 10024 raft_consensus.cc:2932] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index 145 to 220, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:30.927475  7449 catalog_manager.cc:5484] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index 145 to 220, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:30.954810  9318 raft_consensus.cc:2932] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index 145 to 220, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:31.227692  7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ae2ad1ed4448689d8fe8eba7942dd9 with delete_type TABLET_DATA_TOMBSTONED (TS 5d1ee23344854913811909afd87ab724 not found in new config with opid_index 220) from {username='slave'} at 127.0.0.1:37614
I20240503 08:12:31.250974 10119 tablet_replica.cc:330] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: stopping tablet replica
I20240503 08:12:31.251695 10119 raft_consensus.cc:2229] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.252066 10119 raft_consensus.cc:2258] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.254765 10119 ts_tablet_manager.cc:1875] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:12:31.255339  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 46187.4us
I20240503 08:12:31.278416 10119 ts_tablet_manager.cc:1888] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.219
I20240503 08:12:31.278896 10119 log.cc:1196] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 5d1ee23344854913811909afd87ab724: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/91ae2ad1ed4448689d8fe8eba7942dd9
I20240503 08:12:31.280382  7437 catalog_manager.cc:4830] TS 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): tablet 91ae2ad1ed4448689d8fe8eba7942dd9 (table rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]) successfully deleted
W20240503 08:12:31.640352  7660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 111015us
I20240503 08:12:32.002620  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.027984  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.042083  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:32.159150  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.199589  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.255463  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:32.556115  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:12:32.744870  7658 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:40273, user_credentials={real_user=slave}} blocked reactor thread for 76089.9us
I20240503 08:12:33.654527  7591 consensus_queue.cc:237] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:33.666572  8918 raft_consensus.cc:1261] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20240503 08:12:33.669991  9638 consensus_queue.cc:1035] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240503 08:12:33.673370  9638 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 113 to 218, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:33.678939  7445 catalog_manager.cc:5484] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 reported cstate change: config changed from index 113 to 218, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:33.680383  8918 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 113 to 218, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:33.813248  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet 65167136cc2d478cb0efd6a374839c48 with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 218) from {username='slave'} at 127.0.0.1:46506
I20240503 08:12:33.844584 10221 tablet_replica.cc:330] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:12:33.852456 10221 raft_consensus.cc:2229] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.857393 10221 raft_consensus.cc:2258] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.859993 10221 ts_tablet_manager.cc:1875] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:33.877511 10221 ts_tablet_manager.cc:1888] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.217
I20240503 08:12:33.880923 10221 log.cc:1196] T 65167136cc2d478cb0efd6a374839c48 P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/65167136cc2d478cb0efd6a374839c48
I20240503 08:12:33.883075  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet 65167136cc2d478cb0efd6a374839c48 (table rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]) successfully deleted
W20240503 08:12:34.446295  7658 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:40273, user_credentials={real_user=slave}} blocked reactor thread for 58104.7us
I20240503 08:12:35.100131  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:35.138269  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:35.170122  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:35.181344  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:35.268818  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:35.386765  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:35.518833  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:12:35.952001  7787 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 69248.6us
W20240503 08:12:36.212007  7527 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:39615, user_credentials={real_user=slave}} blocked reactor thread for 44503.6us
I20240503 08:12:36.611020  7852 consensus_queue.cc:237] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:36.654274  9317 raft_consensus.cc:1261] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240503 08:12:36.665114 10090 consensus_queue.cc:1035] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 270, Time since last communication: 0.000s
I20240503 08:12:36.675873 10090 raft_consensus.cc:2932] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 167 to 272, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:36.691815  7446 catalog_manager.cc:5484] T a629b67231f84cdbb5e302265896e25a P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index 167 to 272, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:36.713431  9317 raft_consensus.cc:2932] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 167 to 272, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:37.281258  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet a629b67231f84cdbb5e302265896e25a with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 272) from {username='slave'} at 127.0.0.1:46506
I20240503 08:12:37.337548 10341 tablet_replica.cc:330] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:12:37.337873 10341 raft_consensus.cc:2229] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.338142 10341 pending_rounds.cc:62] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Trying to abort 1 pending ops.
I20240503 08:12:37.338336 10341 pending_rounds.cc:68] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Aborting op as it isn't in flight: id { term: 1 index: 271 } timestamp: 7023509325846638592 op_type: WRITE_OP write_request { tablet_id: "a629b67231f84cdbb5e302265896e25a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\214N_\007Y\352\204r\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""38d788039c4575e0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509325747515392 authz_token { token_data: "\010\214\273\322\261\006\"3\n\005slave\022*\n 9822a155357b4da38dc52ebeab8687aa\020\001\030\001 \001(\001" signature: "<redacted>""\272<P\252\003\256\314\337\253\'\356/\364\177#\272\217\275\226uCJ\267N\260k>+z\247\016l\r]~\324y\360!O\022*8\025Jj_\255\2655\200\350\264y^h\005\311\352\261{~\227\374" signing_key_seq_num: 0 } } request_id { client_id: "c2bd9a9119ac4e019293ce97a31eee68" seq_no: 789 first_incomplete_seq_no: 789 attempt_no: 0 }
I20240503 08:12:37.351940 10341 raft_consensus.cc:2258] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.354698 10341 ts_tablet_manager.cc:1875] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:37.383709 10341 ts_tablet_manager.cc:1888] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.271
I20240503 08:12:37.383971 10341 log.cc:1196] T a629b67231f84cdbb5e302265896e25a P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/a629b67231f84cdbb5e302265896e25a
I20240503 08:12:37.385108  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet a629b67231f84cdbb5e302265896e25a (table rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]) successfully deleted
I20240503 08:12:37.940407  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:12:38.039342  7660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 73125us
I20240503 08:12:38.252511  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:38.365154  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:38.395975  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:38.474001  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:38.542513  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:38.564746  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:39.537058  7851 consensus_queue.cc:237] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 1.323, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:39.553476  9315 raft_consensus.cc:1261] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 323. Preceding OpId from leader: term: 1 index: 324. (index mismatch)
I20240503 08:12:39.555931 10089 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20240503 08:12:39.563719 10173 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.324: config changed from index 103 to 324, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:39.571099  7447 catalog_manager.cc:5484] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index 103 to 324, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:39.574335  9318 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Committing config change with OpId 1.324: config changed from index 103 to 324, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:39.609468  7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 303fded976c342efbccfc375950f78ec with delete_type TABLET_DATA_TOMBSTONED (TS 55fc889503854003801521bf97120179 not found in new config with opid_index 324) from {username='slave'} at 127.0.0.1:48186
I20240503 08:12:39.610543 10425 tablet_replica.cc:330] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: stopping tablet replica
I20240503 08:12:39.611053 10425 raft_consensus.cc:2229] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.611515 10425 raft_consensus.cc:2258] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.613603 10425 ts_tablet_manager.cc:1875] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.640527 10425 ts_tablet_manager.cc:1888] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.323
I20240503 08:12:39.640820 10425 log.cc:1196] T 303fded976c342efbccfc375950f78ec P 55fc889503854003801521bf97120179: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/303fded976c342efbccfc375950f78ec
I20240503 08:12:39.641846  7438 catalog_manager.cc:4830] TS 55fc889503854003801521bf97120179 (127.2.49.129:39639): tablet 303fded976c342efbccfc375950f78ec (table rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]) successfully deleted
W20240503 08:12:40.123804  7787 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 141766us
I20240503 08:12:40.285198  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:40.307763  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:40.325300  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:40.352289  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:40.366609  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:40.369195  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:40.591641  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:12:41.187340  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:43861, user_credentials={real_user=slave}} blocked reactor thread for 48301.7us
I20240503 08:12:41.757886  7851 consensus_queue.cc:237] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 410, Committed index: 410, Last appended: 1.410, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:41.760716  9189 raft_consensus.cc:1261] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 410. Preceding OpId from leader: term: 1 index: 411. (index mismatch)
I20240503 08:12:41.762053 10509 consensus_queue.cc:1035] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 411, Last known committed idx: 410, Time since last communication: 0.000s
I20240503 08:12:41.770148 10509 raft_consensus.cc:2932] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.411: config changed from index 65 to 411, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:41.773778  9186 raft_consensus.cc:2932] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Committing config change with OpId 1.411: config changed from index 65 to 411, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:41.780241  7447 catalog_manager.cc:5484] T 7e5a34b052e94b7b8a1dbaec2038d921 P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index 65 to 411, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:41.835016  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5a34b052e94b7b8a1dbaec2038d921 with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 411) from {username='slave'} at 127.0.0.1:46506
I20240503 08:12:41.886885 10527 tablet_replica.cc:330] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:12:41.887282 10527 raft_consensus.cc:2229] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.887635 10527 raft_consensus.cc:2258] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.889003 10527 ts_tablet_manager.cc:1875] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:41.900054 10527 ts_tablet_manager.cc:1888] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.410
I20240503 08:12:41.902725 10527 log.cc:1196] T 7e5a34b052e94b7b8a1dbaec2038d921 P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e5a34b052e94b7b8a1dbaec2038d921
I20240503 08:12:41.904541  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet 7e5a34b052e94b7b8a1dbaec2038d921 (table rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]) successfully deleted
I20240503 08:12:42.890081  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:42.908428  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:42.911085  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:42.941462  7571 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:42.952873  8900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:43.130412  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:43.272112  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:43.630354  7592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0246c726f68848a7ad4620e533f93b53"
dest_uuid: "55fc889503854003801521bf97120179"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54042
I20240503 08:12:43.630791  7592 raft_consensus.cc:595] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:43.900326 10354 raft_consensus.cc:979] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: : Instructing follower fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:12:43.900537 10354 raft_consensus.cc:1067] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Signalling peer fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:12:43.902678  8920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0246c726f68848a7ad4620e533f93b53"
dest_uuid: "fb22180119ec428393402ddbd0b748b6"
 from {username='slave'} at 127.2.49.129:58271
I20240503 08:12:43.903007  8920 raft_consensus.cc:482] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:43.903159  8920 raft_consensus.cc:3037] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:43.912851  8920 raft_consensus.cc:504] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:43.915525  8920 leader_election.cc:288] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 2 election: Requested vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639), 1456c568b524465bb505956695d13738 (127.2.49.133:39615)
I20240503 08:12:43.916596  7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0246c726f68848a7ad4620e533f93b53" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:12:43.916874  7592 raft_consensus.cc:3032] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:43.916970  7592 raft_consensus.cc:729] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:43.917344  7592 consensus_queue.cc:260] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:43.917840  7592 raft_consensus.cc:3037] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:43.919838  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 54346.3us
I20240503 08:12:43.927918  7592 raft_consensus.cc:2454] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 2.
I20240503 08:12:43.934792  8853 leader_election.cc:302] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, fb22180119ec428393402ddbd0b748b6; no voters: 
I20240503 08:12:43.941022 10607 raft_consensus.cc:2781] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:43.948112 10607 raft_consensus.cc:686] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 2 LEADER]: Becoming Leader. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Running, Role: LEADER
I20240503 08:12:43.948762 10607 consensus_queue.cc:237] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:43.961329  7447 catalog_manager.cc:5484] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 reported cstate change: term changed from 1 to 2, leader changed from 55fc889503854003801521bf97120179 (127.2.49.129) to fb22180119ec428393402ddbd0b748b6 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:44.017014  9057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0246c726f68848a7ad4620e533f93b53" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1456c568b524465bb505956695d13738"
I20240503 08:12:44.017395  9057 raft_consensus.cc:3037] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:44.020687  9057 raft_consensus.cc:2454] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 2.
I20240503 08:12:44.330288  9057 raft_consensus.cc:1261] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:12:44.331293 10607 consensus_queue.cc:1035] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:44.382917  7591 raft_consensus.cc:1261] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:12:44.383904 10608 consensus_queue.cc:1035] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.208644  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:45.219920  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:45.271525  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:45.338181  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:45.346693  7571 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:45.473484  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:45.617946  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:46.240456  8917 consensus_queue.cc:237] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:46.301990  9057 raft_consensus.cc:1261] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:12:46.303954 10608 consensus_queue.cc:1035] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.327935 10697 raft_consensus.cc:2932] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:46.331959  9057 raft_consensus.cc:2932] T 0246c726f68848a7ad4620e533f93b53 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:46.336519  7447 catalog_manager.cc:5484] T 0246c726f68848a7ad4620e533f93b53 P fb22180119ec428393402ddbd0b748b6 reported cstate change: config changed from index 2 to 4, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:46.830828  7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 0246c726f68848a7ad4620e533f93b53 with delete_type TABLET_DATA_TOMBSTONED (TS 55fc889503854003801521bf97120179 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48186
I20240503 08:12:46.914052 10720 tablet_replica.cc:330] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: stopping tablet replica
I20240503 08:12:46.917081 10720 raft_consensus.cc:2229] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.920852 10720 raft_consensus.cc:2258] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.925977 10720 ts_tablet_manager.cc:1875] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.964066 10720 ts_tablet_manager.cc:1888] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:12:46.964346 10720 log.cc:1196] T 0246c726f68848a7ad4620e533f93b53 P 55fc889503854003801521bf97120179: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/0246c726f68848a7ad4620e533f93b53
I20240503 08:12:46.965390  7438 catalog_manager.cc:4830] TS 55fc889503854003801521bf97120179 (127.2.49.129:39639): tablet 0246c726f68848a7ad4620e533f93b53 (table TestTable [id=7d99db23369f4ce98f11aa4d899291b3]) successfully deleted
W20240503 08:12:47.076527  8852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:39615, user_credentials={real_user=slave}} blocked reactor thread for 60895.6us
I20240503 08:12:47.774247  9039 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:47.789440  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:47.789680  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:47.902215  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:47.997751  7571 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:48.187095  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:48.246994  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:48.649641  7591 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f5a650ed26a406baa15105ac9fc8f8f"
dest_uuid: "55fc889503854003801521bf97120179"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54076
I20240503 08:12:48.649870  7591 raft_consensus.cc:595] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:12:48.662789  7552 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.667308  7552 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.680847  7552 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.687423  7552 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.699225  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.717463  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.726529  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.737874  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.751520  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.770679  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.788690  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.806311  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.823828  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.847702  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.867691  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.889355  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.912509  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.959978  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:12:48.986543  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
I20240503 08:12:49.037950 10386 raft_consensus.cc:979] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: : Instructing follower 1456c568b524465bb505956695d13738 to start an election
I20240503 08:12:49.038122 10386 raft_consensus.cc:1067] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 LEADER]: Signalling peer 1456c568b524465bb505956695d13738 to start an election
I20240503 08:12:49.039230  9057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f5a650ed26a406baa15105ac9fc8f8f"
dest_uuid: "1456c568b524465bb505956695d13738"
 from {username='slave'} at 127.2.49.129:42755
I20240503 08:12:49.039566  9057 raft_consensus.cc:482] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:49.039718  9057 raft_consensus.cc:3037] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:49.040851  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
I20240503 08:12:49.047536  9057 raft_consensus.cc:504] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:49.052250  7851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5a650ed26a406baa15105ac9fc8f8f" candidate_uuid: "1456c568b524465bb505956695d13738" candidate_term: 2 candidate_status { last_received { term: 1 index: 424 } } ignore_live_leader: true dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:12:49.052479  7851 raft_consensus.cc:3037] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:49.052824  7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5a650ed26a406baa15105ac9fc8f8f" candidate_uuid: "1456c568b524465bb505956695d13738" candidate_term: 2 candidate_status { last_received { term: 1 index: 424 } } ignore_live_leader: true dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:12:49.053022  7592 raft_consensus.cc:3032] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:49.053090  7592 raft_consensus.cc:729] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:12:49.053308  7592 consensus_queue.cc:260] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 1.424, Last appended by leader: 424, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:49.053647  7592 raft_consensus.cc:3037] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:49.054996  9057 leader_election.cc:288] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [CANDIDATE]: Term 2 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273), 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:12:49.055200  7851 raft_consensus.cc:2454] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1456c568b524465bb505956695d13738 in term 2.
I20240503 08:12:49.056285  7592 raft_consensus.cc:2454] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1456c568b524465bb505956695d13738 in term 2.
I20240503 08:12:49.056805  8993 leader_election.cc:302] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1456c568b524465bb505956695d13738, 5d1ee23344854913811909afd87ab724; no voters: 
W20240503 08:12:49.064800  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Illegal state: replica 55fc889503854003801521bf97120179 is not leader of this config: current role FOLLOWER
I20240503 08:12:49.070478 10812 raft_consensus.cc:2781] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:49.081935 10812 raft_consensus.cc:686] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 2 LEADER]: Becoming Leader. State: Replica: 1456c568b524465bb505956695d13738, State: Running, Role: LEADER
I20240503 08:12:49.082688 10812 consensus_queue.cc:237] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 1.424, Last appended by leader: 424, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:49.108466  7446 catalog_manager.cc:5484] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 reported cstate change: term changed from 1 to 2, leader changed from 55fc889503854003801521bf97120179 (127.2.49.129) to 1456c568b524465bb505956695d13738 (127.2.49.133). New cstate: current_term: 2 leader_uuid: "1456c568b524465bb505956695d13738" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
W20240503 08:12:49.196435  7812 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39506: Illegal state: replica 5d1ee23344854913811909afd87ab724 is not leader of this config: current role FOLLOWER
I20240503 08:12:49.263842  7592 raft_consensus.cc:1261] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Refusing update from remote peer 1456c568b524465bb505956695d13738: Log matching property violated. Preceding OpId in replica: term: 1 index: 424. Preceding OpId from leader: term: 2 index: 426. (index mismatch)
I20240503 08:12:49.263906  7852 raft_consensus.cc:1261] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Refusing update from remote peer 1456c568b524465bb505956695d13738: Log matching property violated. Preceding OpId in replica: term: 1 index: 424. Preceding OpId from leader: term: 2 index: 426. (index mismatch)
I20240503 08:12:49.264678 10812 consensus_queue.cc:1035] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I20240503 08:12:49.265714 10812 consensus_queue.cc:1035] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I20240503 08:12:49.269999 10762 mvcc.cc:205] Tried to move back new op lower bound from 7023509378097471488 to 7023509377452994560. Current Snapshot: MvccSnapshot[applied={T|T < 7023509378097471488 or (T in {7023509378097471488})}]
I20240503 08:12:49.277647 10770 mvcc.cc:205] Tried to move back new op lower bound from 7023509378097471488 to 7023509377452994560. Current Snapshot: MvccSnapshot[applied={T|T < 7023509378097471488}]
I20240503 08:12:49.637014  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:49.642994  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:49.664417  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:49.729007  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:49.748195  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:49.801765  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:49.976867  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:50.881398  9057 consensus_queue.cc:237] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 449, Committed index: 449, Last appended: 2.450, Last appended by leader: 424, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 451 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:12:50.889935  7852 raft_consensus.cc:1261] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Refusing update from remote peer 1456c568b524465bb505956695d13738: Log matching property violated. Preceding OpId in replica: term: 2 index: 449. Preceding OpId from leader: term: 2 index: 451. (index mismatch)
I20240503 08:12:50.891157 10812 consensus_queue.cc:1035] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 451, Last known committed idx: 449, Time since last communication: 0.000s
I20240503 08:12:50.904526 10813 raft_consensus.cc:2932] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 [term 2 LEADER]: Committing config change with OpId 2.451: config changed from index 56 to 451, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 451 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:50.911855  7851 raft_consensus.cc:2932] T 9f5a650ed26a406baa15105ac9fc8f8f P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Committing config change with OpId 2.451: config changed from index 56 to 451, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 451 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:50.915081  7447 catalog_manager.cc:5484] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 reported cstate change: config changed from index 56 to 451, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "1456c568b524465bb505956695d13738" committed_config { opid_index: 451 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:12:51.095889  7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5a650ed26a406baa15105ac9fc8f8f with delete_type TABLET_DATA_TOMBSTONED (TS 55fc889503854003801521bf97120179 not found in new config with opid_index 451) from {username='slave'} at 127.0.0.1:48186
I20240503 08:12:51.113446 10902 tablet_replica.cc:330] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: stopping tablet replica
I20240503 08:12:51.113812 10902 raft_consensus.cc:2229] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.114197 10902 raft_consensus.cc:2258] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.115339 10902 ts_tablet_manager.cc:1875] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:51.145809 10902 ts_tablet_manager.cc:1888] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.450
I20240503 08:12:51.147891 10902 log.cc:1196] T 9f5a650ed26a406baa15105ac9fc8f8f P 55fc889503854003801521bf97120179: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f5a650ed26a406baa15105ac9fc8f8f
I20240503 08:12:51.149106  7438 catalog_manager.cc:4830] TS 55fc889503854003801521bf97120179 (127.2.49.129:39639): tablet 9f5a650ed26a406baa15105ac9fc8f8f (table rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]) successfully deleted
W20240503 08:12:51.873456  7787 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 135346us
I20240503 08:12:52.748344  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:52.853116  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:52.857312  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:52.901743  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:12:53.010404  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 63076.5us
W20240503 08:12:53.017473  7787 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 71748.3us
I20240503 08:12:53.092463  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:53.165004  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:53.205631  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:53.913875  7851 consensus_queue.cc:237] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 1.497, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:12:53.953387  9185 raft_consensus.cc:1261] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
W20240503 08:12:53.956468  8852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:39615, user_credentials={real_user=slave}} blocked reactor thread for 95767.7us
I20240503 08:12:53.963565 10777 consensus_queue.cc:1035] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 08:12:54.009835 10947 raft_consensus.cc:2932] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.498: config changed from index 128 to 498, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:54.012486  9189 raft_consensus.cc:2932] T 44c94e75cff640f8927043b5e071c980 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index 128 to 498, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:54.017576  7447 catalog_manager.cc:5484] T 44c94e75cff640f8927043b5e071c980 P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index 128 to 498, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:12:54.058211  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c94e75cff640f8927043b5e071c980 with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 498) from {username='slave'} at 127.0.0.1:46506
I20240503 08:12:54.088603 11015 tablet_replica.cc:330] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:12:54.088919 11015 raft_consensus.cc:2229] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.089151 11015 pending_rounds.cc:62] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Trying to abort 1 pending ops.
I20240503 08:12:54.089232 11015 pending_rounds.cc:68] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Aborting op as it isn't in flight: id { term: 1 index: 497 } timestamp: 7023509397107687424 op_type: WRITE_OP write_request { tablet_id: "44c94e75cff640f8927043b5e071c980" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000H\027\374\010K\307\366\211\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0a3d10d85e39b450" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509396952924160 authz_token { token_data: "\010\215\273\322\261\006\"3\n\005slave\022*\n cb95e97e3d554c499c162fbea5b8f2f5\020\001\030\001 \001(\001" signature: "<redacted>""#\226C}r\316l\0045\322z\313n\267\354\013*X{\330\236\017\264!\225t\265\240\263\0360\330\005Sz\336\204c\205W\r\373K\340\213\375\205J@\014\257L9}D$\305\340\224_q\220#H" signing_key_seq_num: 0 } } request_id { client_id: "1db0bf261d5648448369a32b00355855" seq_no: 1502 first_incomplete_seq_no: 1502 attempt_no: 0 }
I20240503 08:12:54.090018 11015 raft_consensus.cc:2258] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.091419 11015 ts_tablet_manager.cc:1875] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.125564 11015 ts_tablet_manager.cc:1888] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.497
I20240503 08:12:54.125799 11015 log.cc:1196] T 44c94e75cff640f8927043b5e071c980 P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/44c94e75cff640f8927043b5e071c980
I20240503 08:12:54.127080  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet 44c94e75cff640f8927043b5e071c980 (table rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]) successfully deleted
W20240503 08:12:54.991695  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 49935.9us
I20240503 08:12:55.365397  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:55.592561  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:55.611577  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:55.630697  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:55.650415  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:55.654287  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:55.742822  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:12:55.816876  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 49371.1us
W20240503 08:12:55.947458  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 47509.4us
I20240503 08:12:56.602371  7592 consensus_queue.cc:237] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 564, Committed index: 564, Last appended: 1.564, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:12:56.612406  9319 raft_consensus.cc:1261] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 564. Preceding OpId from leader: term: 1 index: 565. (index mismatch)
I20240503 08:12:56.613200 10925 consensus_queue.cc:1035] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 565, Last known committed idx: 564, Time since last communication: 0.000s
I20240503 08:12:56.615780 10925 raft_consensus.cc:2932] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.565: config changed from index 74 to 565, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:56.625120  9316 raft_consensus.cc:2932] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Committing config change with OpId 1.565: config changed from index 74 to 565, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:56.627238  7447 catalog_manager.cc:5484] T 6c6b0211f6b44c1b92a5be884608d000 P 55fc889503854003801521bf97120179 reported cstate change: config changed from index 74 to 565, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } }
I20240503 08:12:56.668452  7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c6b0211f6b44c1b92a5be884608d000 with delete_type TABLET_DATA_TOMBSTONED (TS 5d1ee23344854913811909afd87ab724 not found in new config with opid_index 565) from {username='slave'} at 127.0.0.1:37614
I20240503 08:12:56.690904 11102 tablet_replica.cc:330] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: stopping tablet replica
I20240503 08:12:56.693486 11102 raft_consensus.cc:2229] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:56.693868 11102 pending_rounds.cc:62] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Trying to abort 1 pending ops.
I20240503 08:12:56.694036 11102 pending_rounds.cc:68] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Aborting op as it isn't in flight: id { term: 1 index: 563 } timestamp: 7023509408060133376 op_type: WRITE_OP write_request { tablet_id: "6c6b0211f6b44c1b92a5be884608d000" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000H\003\222&\0215v\'\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""2fda50b7cbd2af7f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509408002125824 authz_token { token_data: "\010\215\273\322\261\006\"3\n\005slave\022*\n 2bb106e82ba64c3b8ac8cc46227b30a9\020\001\030\001 \001(\001" signature: "<redacted>""d\n\030\330\246W\344\272\303KbUQ\320e\207\240\263\355\244\366?\232\256y\035+{XRs)\306\256\305\301\013;V\377\315\234\310\026\260=\000-\nmZ5\326\264\020Y]\210\246{\2142\254C" signing_key_seq_num: 0 } } request_id { client_id: "ed172f58cd294a18ab935896391d9aea" seq_no: 1649 first_incomplete_seq_no: 1649 attempt_no: 0 }
I20240503 08:12:56.698457 11102 raft_consensus.cc:2258] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:56.699638 11102 ts_tablet_manager.cc:1875] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:56.722132 11102 ts_tablet_manager.cc:1888] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.563
I20240503 08:12:56.722401 11102 log.cc:1196] T 6c6b0211f6b44c1b92a5be884608d000 P 5d1ee23344854913811909afd87ab724: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/6c6b0211f6b44c1b92a5be884608d000
I20240503 08:12:56.723359  7437 catalog_manager.cc:4830] TS 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): tablet 6c6b0211f6b44c1b92a5be884608d000 (table rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]) successfully deleted
I20240503 08:12:57.657809  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:57.662621  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:12:57.674808  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:43861, user_credentials={real_user=slave}} blocked reactor thread for 43708.6us
I20240503 08:12:57.698184  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:57.742255  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:57.763829  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:57.782243  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:57.791596  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:58.251132  7723 consensus_queue.cc:237] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 685, Committed index: 685, Last appended: 1.685, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 686 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:12:58.257210  8917 raft_consensus.cc:1261] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 685. Preceding OpId from leader: term: 1 index: 686. (index mismatch)
I20240503 08:12:58.259106 11184 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 686, Last known committed idx: 685, Time since last communication: 0.000s
I20240503 08:12:58.266603 10628 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.686: config changed from index 154 to 686, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 686 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:58.280506  7446 catalog_manager.cc:5484] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index 154 to 686, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 686 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:58.293736  8920 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Committing config change with OpId 1.686: config changed from index 154 to 686, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 686 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:12:58.390807  7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f87cf2ca7041cd8727e41d994f330c with delete_type TABLET_DATA_TOMBSTONED (TS 5d1ee23344854913811909afd87ab724 not found in new config with opid_index 686) from {username='slave'} at 127.0.0.1:37614
I20240503 08:12:58.443495 11195 tablet_replica.cc:330] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: stopping tablet replica
I20240503 08:12:58.443856 11195 raft_consensus.cc:2229] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.445403 11195 pending_rounds.cc:62] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Trying to abort 1 pending ops.
I20240503 08:12:58.445526 11195 pending_rounds.cc:68] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Aborting op as it isn't in flight: id { term: 1 index: 685 } timestamp: 7023509414834966528 op_type: WRITE_OP write_request { tablet_id: "01f87cf2ca7041cd8727e41d994f330c" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000)\315\347+\000\343\277\357\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bee2dbd23f9bb07e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509414719610880 authz_token { token_data: "\010\215\273\322\261\006\"3\n\005slave\022*\n 1610d4e6b14e452080592c64036ae9e0\020\001\030\001 \001(\001" signature: "<redacted>""+\245\323\353\275sg\2155\212\225\0201P&L\001\302\300\023*\034\301K\350\006\030\024\371\277\003?|\213vK\320\035\353\214\240H\033i\235\371K\364Q;Z\247\\\030\340\'\250\\\271\2701p\343\005" signing_key_seq_num: 0 } } request_id { client_id: "00429460b7d04a85993ebb57d5a8fda6" seq_no: 2024 first_incomplete_seq_no: 2024 attempt_no: 0 }
I20240503 08:12:58.449337 11195 raft_consensus.cc:2258] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.451659 11195 ts_tablet_manager.cc:1875] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:58.489656 11195 ts_tablet_manager.cc:1888] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.685
I20240503 08:12:58.491179 11195 log.cc:1196] T 01f87cf2ca7041cd8727e41d994f330c P 5d1ee23344854913811909afd87ab724: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/01f87cf2ca7041cd8727e41d994f330c
I20240503 08:12:58.493122  7437 catalog_manager.cc:4830] TS 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): tablet 01f87cf2ca7041cd8727e41d994f330c (table rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]) successfully deleted
I20240503 08:12:59.423419  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.452093  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:59.474188  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:59.548724  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:59.578702  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:59.641400  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:59.658849  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:00.414124  7852 consensus_queue.cc:237] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 593, Committed index: 593, Last appended: 1.593, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:00.416148  9059 raft_consensus.cc:1261] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 593. Preceding OpId from leader: term: 1 index: 594. (index mismatch)
I20240503 08:13:00.416882 11272 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 594, Last known committed idx: 593, Time since last communication: 0.000s
I20240503 08:13:00.428136 11272 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.594: config changed from index 107 to 594, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:00.431138  9059 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Committing config change with OpId 1.594: config changed from index 107 to 594, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:00.437232  7445 catalog_manager.cc:5484] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 reported cstate change: config changed from index 107 to 594, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 594 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:00.675184  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet 6462c70bafc94b7fb3bcb9502e80e096 with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 594) from {username='slave'} at 127.0.0.1:46506
I20240503 08:13:00.680117 11293 tablet_replica.cc:330] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:13:00.680354 11293 raft_consensus.cc:2229] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:00.680562 11293 raft_consensus.cc:2258] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:00.686390 11293 ts_tablet_manager.cc:1875] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:00.709820 11293 ts_tablet_manager.cc:1888] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.593
I20240503 08:13:00.713716 11293 log.cc:1196] T 6462c70bafc94b7fb3bcb9502e80e096 P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:00.717329  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet 6462c70bafc94b7fb3bcb9502e80e096 (table rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]) successfully deleted
I20240503 08:13:01.713907  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.760504  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:01.785941  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:01.797708  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:01.835347  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:01.873926  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:01.943354  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:02.309091  7592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00"
dest_uuid: "55fc889503854003801521bf97120179"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:58560
I20240503 08:13:02.309353  7592 raft_consensus.cc:595] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:13:02.317277  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.333673  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.342088  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.354470  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.369705  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.379863  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.416046  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.432112  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.447432  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.461098  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.473966  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.490247  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.505614  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.521008  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.540468  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.561967  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.582089  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.605985  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.626642  7551 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.655016  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.707295  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.732316  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.758288  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.784108  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.813660  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
I20240503 08:13:02.828014 11312 raft_consensus.cc:979] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: : Instructing follower 359f2f323e79431eac07dc0a3f47a47b to start an election
I20240503 08:13:02.828198 11312 raft_consensus.cc:1067] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Signalling peer 359f2f323e79431eac07dc0a3f47a47b to start an election
I20240503 08:13:02.830001  7721 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00"
dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
 from {username='slave'} at 127.2.49.129:39989
I20240503 08:13:02.835996  7721 raft_consensus.cc:482] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:02.837697  7721 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:02.851223  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
W20240503 08:13:02.885413  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Service unavailable: leader transfer in progress
I20240503 08:13:02.889349  7721 raft_consensus.cc:504] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:02.904057  7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 2 candidate_status { last_received { term: 1 index: 645 } } ignore_live_leader: true dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:13:02.904286  7721 leader_election.cc:288] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 2 election: Requested vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639), fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615)
I20240503 08:13:02.904321  7592 raft_consensus.cc:3032] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:02.904412  7592 raft_consensus.cc:729] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:13:02.904670  7592 consensus_queue.cc:260] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 645, Committed index: 645, Last appended: 1.645, Last appended by leader: 645, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:02.904721  8920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00" candidate_uuid: "359f2f323e79431eac07dc0a3f47a47b" candidate_term: 2 candidate_status { last_received { term: 1 index: 645 } } ignore_live_leader: true dest_uuid: "fb22180119ec428393402ddbd0b748b6"
I20240503 08:13:02.904994  7592 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:02.905236  8920 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:02.907765  7592 raft_consensus.cc:2454] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 2.
I20240503 08:13:02.908423  8920 raft_consensus.cc:2454] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f2f323e79431eac07dc0a3f47a47b in term 2.
I20240503 08:13:02.908520  7659 leader_election.cc:302] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, 55fc889503854003801521bf97120179; no voters: 
I20240503 08:13:02.908862 11132 raft_consensus.cc:2781] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:02.909034 11132 raft_consensus.cc:686] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Becoming Leader. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Running, Role: LEADER
I20240503 08:13:02.909359 11132 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 645, Committed index: 645, Last appended: 1.645, Last appended by leader: 645, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:02.915362  7445 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: term changed from 1 to 2, leader changed from 55fc889503854003801521bf97120179 (127.2.49.129) to 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130). New cstate: current_term: 2 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
W20240503 08:13:02.915925  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60574: Illegal state: replica 55fc889503854003801521bf97120179 is not leader of this config: current role FOLLOWER
I20240503 08:13:03.067581  8920 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 645. Preceding OpId from leader: term: 2 index: 647. (index mismatch)
I20240503 08:13:03.068724 11262 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 646, Last known committed idx: 645, Time since last communication: 0.000s
I20240503 08:13:03.079906  7591 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 645. Preceding OpId from leader: term: 2 index: 647. (index mismatch)
I20240503 08:13:03.081189 11389 mvcc.cc:205] Tried to move back new op lower bound from 7023509434639183872 to 7023509433999847424. Current Snapshot: MvccSnapshot[applied={T|T < 7023509434639183872}]
I20240503 08:13:03.081807 11132 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 646, Last known committed idx: 645, Time since last communication: 0.000s
I20240503 08:13:03.168745  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.250937  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:03.253218  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:03.281610  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:03.295902  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:03.323871  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:03.334466  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:04.640051  7721 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 677, Committed index: 677, Last appended: 2.677, Last appended by leader: 645, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:04.642359  8920 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 2 index: 677. Preceding OpId from leader: term: 2 index: 678. (index mismatch)
I20240503 08:13:04.643041 11420 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240503 08:13:04.645747 11420 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Committing config change with OpId 2.678: config changed from index 70 to 678, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:04.646435  8920 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Committing config change with OpId 2.678: config changed from index 70 to 678, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:04.652743  7447 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index 70 to 678, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:04.687141  7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 80eccc47e060493ebe43b007c7a23f00 with delete_type TABLET_DATA_TOMBSTONED (TS 55fc889503854003801521bf97120179 not found in new config with opid_index 678) from {username='slave'} at 127.0.0.1:48186
I20240503 08:13:04.693677 11462 tablet_replica.cc:330] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: stopping tablet replica
I20240503 08:13:04.693986 11462 raft_consensus.cc:2229] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.694286 11462 raft_consensus.cc:2258] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.699563 11462 ts_tablet_manager.cc:1875] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.707875 11462 ts_tablet_manager.cc:1888] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.677
I20240503 08:13:04.708070 11462 log.cc:1196] T 80eccc47e060493ebe43b007c7a23f00 P 55fc889503854003801521bf97120179: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/80eccc47e060493ebe43b007c7a23f00
I20240503 08:13:04.709316  7438 catalog_manager.cc:4830] TS 55fc889503854003801521bf97120179 (127.2.49.129:39639): tablet 80eccc47e060493ebe43b007c7a23f00 (table rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]) successfully deleted
I20240503 08:13:05.414897  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.453963  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:05.481413  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:05.535771  7834 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:05.655345  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:05.662317  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:05.752692  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:06.218899  7851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8967d6dac1aa482683c8bad1892dcbde"
dest_uuid: "5d1ee23344854913811909afd87ab724"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54698
I20240503 08:13:06.219180  7851 raft_consensus.cc:595] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:06.221292 11271 raft_consensus.cc:979] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: : Instructing follower 55fc889503854003801521bf97120179 to start an election
I20240503 08:13:06.221423 11271 raft_consensus.cc:1067] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Signalling peer 55fc889503854003801521bf97120179 to start an election
W20240503 08:13:06.264079  7814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56522: Service unavailable: leader transfer in progress
W20240503 08:13:06.268250  7814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56522: Service unavailable: leader transfer in progress
I20240503 08:13:06.273787  7593 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8967d6dac1aa482683c8bad1892dcbde"
dest_uuid: "55fc889503854003801521bf97120179"
 from {username='slave'} at 127.2.49.131:33349
I20240503 08:13:06.274107  7593 raft_consensus.cc:482] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:06.274266  7593 raft_consensus.cc:3037] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:06.277475  7593 raft_consensus.cc:504] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:06.279790  7593 leader_election.cc:288] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 2 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273), b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149)
W20240503 08:13:06.279960  7812 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56522: Service unavailable: leader transfer in progress
I20240503 08:13:06.281299  7851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8967d6dac1aa482683c8bad1892dcbde" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 2 candidate_status { last_received { term: 1 index: 700 } } ignore_live_leader: true dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:13:06.281528  7851 raft_consensus.cc:3032] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:06.281620  7851 raft_consensus.cc:729] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:13:06.281916  7851 consensus_queue.cc:260] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 700, Committed index: 700, Last appended: 1.700, Last appended by leader: 700, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:06.282481  7851 raft_consensus.cc:3037] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:06.286633  7851 raft_consensus.cc:2454] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc889503854003801521bf97120179 in term 2.
W20240503 08:13:06.287003 11271 consensus_queue.cc:1159] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "55fc889503854003801521bf97120179" responder_term: 1 status { last_received { term: 1 index: 700 } last_committed_idx: 700 last_received_current_leader { term: 1 index: 700 } }
I20240503 08:13:06.290403  7527 leader_election.cc:302] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fc889503854003801521bf97120179, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:13:06.290755 11125 raft_consensus.cc:2781] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:06.290907 11125 raft_consensus.cc:686] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 2 LEADER]: Becoming Leader. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:13:06.291184 11125 consensus_queue.cc:237] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 700, Committed index: 700, Last appended: 1.700, Last appended by leader: 700, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:06.294579  7447 catalog_manager.cc:5484] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 reported cstate change: term changed from 1 to 2, leader changed from 5d1ee23344854913811909afd87ab724 (127.2.49.131) to 55fc889503854003801521bf97120179 (127.2.49.129). New cstate: current_term: 2 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
W20240503 08:13:06.301458  7814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56522: Illegal state: replica 5d1ee23344854913811909afd87ab724 is not leader of this config: current role FOLLOWER
I20240503 08:13:06.317832  7851 raft_consensus.cc:1261] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 700. Preceding OpId from leader: term: 2 index: 702. (index mismatch)
I20240503 08:13:06.319262 11312 consensus_queue.cc:1035] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 701, Last known committed idx: 700, Time since last communication: 0.000s
I20240503 08:13:06.351182  9186 raft_consensus.cc:3037] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:06.352345  9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8967d6dac1aa482683c8bad1892dcbde" candidate_uuid: "55fc889503854003801521bf97120179" candidate_term: 2 candidate_status { last_received { term: 1 index: 700 } } ignore_live_leader: true dest_uuid: "b50f0dfea2284eacbe4184b1f3ae3753"
I20240503 08:13:06.365559  9186 raft_consensus.cc:1261] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 700. Preceding OpId from leader: term: 2 index: 702. (index mismatch)
I20240503 08:13:06.377082 11125 consensus_queue.cc:1035] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 701, Last known committed idx: 700, Time since last communication: 0.000s
W20240503 08:13:06.598902  7526 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 57791.5us
W20240503 08:13:06.853011  7660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 47325us
I20240503 08:13:07.201133  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:07.246454  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:07.261535  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:07.314145  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:07.328271  7834 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:07.349372  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:07.412478  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:07.888371  7593 consensus_queue.cc:237] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 727, Committed index: 727, Last appended: 2.727, Last appended by leader: 700, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 728 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:07.891778  9188 raft_consensus.cc:1261] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 2 index: 727. Preceding OpId from leader: term: 2 index: 728. (index mismatch)
I20240503 08:13:07.892410 11549 consensus_queue.cc:1035] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 728, Last known committed idx: 727, Time since last communication: 0.000s
I20240503 08:13:07.895881 11312 raft_consensus.cc:2932] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 [term 2 LEADER]: Committing config change with OpId 2.728: config changed from index 95 to 728, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 728 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:07.896895  9188 raft_consensus.cc:2932] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Committing config change with OpId 2.728: config changed from index 95 to 728, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 728 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:07.909978  7447 catalog_manager.cc:5484] T 8967d6dac1aa482683c8bad1892dcbde P 55fc889503854003801521bf97120179 reported cstate change: config changed from index 95 to 728, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New cstate: current_term: 2 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 728 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
W20240503 08:13:07.987673  7660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 50631.2us
I20240503 08:13:07.999289  7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 8967d6dac1aa482683c8bad1892dcbde with delete_type TABLET_DATA_TOMBSTONED (TS 5d1ee23344854913811909afd87ab724 not found in new config with opid_index 728) from {username='slave'} at 127.0.0.1:37614
I20240503 08:13:08.085711 11625 tablet_replica.cc:330] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: stopping tablet replica
I20240503 08:13:08.086045 11625 raft_consensus.cc:2229] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:08.130412 11625 raft_consensus.cc:2258] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:08.138865 11625 ts_tablet_manager.cc:1875] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:08.149209 11625 ts_tablet_manager.cc:1888] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.727
I20240503 08:13:08.150504 11625 log.cc:1196] T 8967d6dac1aa482683c8bad1892dcbde P 5d1ee23344854913811909afd87ab724: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/8967d6dac1aa482683c8bad1892dcbde
I20240503 08:13:08.151806  7437 catalog_manager.cc:4830] TS 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): tablet 8967d6dac1aa482683c8bad1892dcbde (table rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]) successfully deleted
I20240503 08:13:10.651696  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:10.652946  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:10.700174  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:10.709497  7834 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:10.761286  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:10.766942  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:10.826857  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:11.384935  7721 consensus_queue.cc:237] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 908, Committed index: 908, Last appended: 1.908, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:11.389990  8919 raft_consensus.cc:1261] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 908. Preceding OpId from leader: term: 1 index: 909. (index mismatch)
I20240503 08:13:11.390862 11548 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 909, Last known committed idx: 908, Time since last communication: 0.000s
I20240503 08:13:11.393783 11548 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.909: config changed from index 686 to 909, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:11.402518  8918 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Committing config change with OpId 1.909: config changed from index 686 to 909, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New config: { opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:11.438951  7447 catalog_manager.cc:5484] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 reported cstate change: config changed from index 686 to 909, VOTER 1456c568b524465bb505956695d13738 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
W20240503 08:13:11.603238  7658 consensus_peers.cc:485] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b -> Peer 1456c568b524465bb505956695d13738 (127.2.49.133:39615): Couldn't send request to peer 1456c568b524465bb505956695d13738. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01f87cf2ca7041cd8727e41d994f330c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:11.758562  7593 consensus_queue.cc:237] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 774, Committed index: 774, Last appended: 1.774, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:11.763897  7721 raft_consensus.cc:1261] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 774. Preceding OpId from leader: term: 1 index: 775. (index mismatch)
W20240503 08:13:11.765470  7529 consensus_peers.cc:485] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7f1cb831a9149e494cb31fac27f922d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:11.774036 11549 consensus_queue.cc:1035] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 775, Last known committed idx: 773, Time since last communication: 0.000s
I20240503 08:13:11.785125 11725 raft_consensus.cc:2932] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.775: config changed from index -1 to 775, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:11.785988  7721 raft_consensus.cc:2932] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Committing config change with OpId 1.775: config changed from index -1 to 775, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New config: { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:11.790606  7447 catalog_manager.cc:5484] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 reported cstate change: config changed from index -1 to 775, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:11.804459 11822 ts_tablet_manager.cc:902] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: Initiating tablet copy from peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:13:11.879278 11822 tablet_copy_client.cc:308] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:43861
I20240503 08:13:11.921054  7744 tablet_copy_service.cc:140] P 359f2f323e79431eac07dc0a3f47a47b: Received BeginTabletCopySession request for tablet 01f87cf2ca7041cd8727e41d994f330c from peer 1456c568b524465bb505956695d13738 ({username='slave'} at 127.2.49.133:40821)
I20240503 08:13:11.921312  7744 tablet_copy_service.cc:161] P 359f2f323e79431eac07dc0a3f47a47b: Beginning new tablet copy session on tablet 01f87cf2ca7041cd8727e41d994f330c from peer 1456c568b524465bb505956695d13738 at {username='slave'} at 127.2.49.133:40821: session id = 1456c568b524465bb505956695d13738-01f87cf2ca7041cd8727e41d994f330c
I20240503 08:13:11.926041  7744 tablet_copy_source_session.cc:205] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:11.932337 11822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f87cf2ca7041cd8727e41d994f330c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.942317 11822 tablet_copy_client.cc:783] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:11.942986 11822 tablet_copy_client.cc:647] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:11.949117 11822 tablet_copy_client.cc:523] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:11.953227 11822 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: Bootstrap starting.
I20240503 08:13:12.307832 11846 ts_tablet_manager.cc:902] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:13:12.437922 11846 tablet_copy_client.cc:308] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:13:12.470329  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet a7f1cb831a9149e494cb31fac27f922d from peer fb22180119ec428393402ddbd0b748b6 ({username='slave'} at 127.2.49.132:37923)
I20240503 08:13:12.470543  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet a7f1cb831a9149e494cb31fac27f922d from peer fb22180119ec428393402ddbd0b748b6 at {username='slave'} at 127.2.49.132:37923: session id = fb22180119ec428393402ddbd0b748b6-a7f1cb831a9149e494cb31fac27f922d
I20240503 08:13:12.473361  7613 tablet_copy_source_session.cc:205] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:12.524412 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f1cb831a9149e494cb31fac27f922d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.543548 11846 tablet_copy_client.cc:783] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:12.545933 11846 tablet_copy_client.cc:647] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:12.558976 11846 tablet_copy_client.cc:523] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:13:12.562350  8852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:39615, user_credentials={real_user=slave}} blocked reactor thread for 70146.2us
I20240503 08:13:12.563139 11846 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: Bootstrap starting.
I20240503 08:13:12.623896  7852 consensus_queue.cc:237] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 810, Committed index: 810, Last appended: 1.810, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:12.627467  9057 raft_consensus.cc:1261] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 810. Preceding OpId from leader: term: 1 index: 811. (index mismatch)
W20240503 08:13:12.627895  7787 consensus_peers.cc:485] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 -> Peer b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149): Couldn't send request to peer b50f0dfea2284eacbe4184b1f3ae3753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6462c70bafc94b7fb3bcb9502e80e096. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.628341 11272 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 811, Last known committed idx: 810, Time since last communication: 0.000s
I20240503 08:13:12.631495 11440 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.811: config changed from index 594 to 811, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:12.635054  9057 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Committing config change with OpId 1.811: config changed from index 594 to 811, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New config: { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:12.642482  7446 catalog_manager.cc:5484] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 reported cstate change: config changed from index 594 to 811, VOTER b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:13.285804 11883 ts_tablet_manager.cc:902] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:13:13.409173 11883 tablet_copy_client.cc:308] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:13:13.409976  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 6462c70bafc94b7fb3bcb9502e80e096 from peer b50f0dfea2284eacbe4184b1f3ae3753 ({username='slave'} at 127.2.49.134:41109)
I20240503 08:13:13.410125  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 6462c70bafc94b7fb3bcb9502e80e096 from peer b50f0dfea2284eacbe4184b1f3ae3753 at {username='slave'} at 127.2.49.134:41109: session id = b50f0dfea2284eacbe4184b1f3ae3753-6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:13.412919  7874 tablet_copy_source_session.cc:205] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:13.459075 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6462c70bafc94b7fb3bcb9502e80e096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.623978 11883 tablet_copy_client.cc:783] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:13.624732 11883 tablet_copy_client.cc:647] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:13.701115 11883 tablet_copy_client.cc:523] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:13.705247 11883 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap starting.
I20240503 08:13:13.983529  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.084128  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:14.139833  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:14.210898  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:14.227427  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:14.253003  7834 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:13:14.318900  7658 consensus_peers.cc:485] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b -> Peer 1456c568b524465bb505956695d13738 (127.2.49.133:39615): Couldn't send request to peer 1456c568b524465bb505956695d13738. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:14.408977 11822 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: Bootstrap replayed 1/1 log segments. Stats: ops{read=913 overwritten=0 applied=912 ignored=0} inserts{seen=1541 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:13:14.409808 11822 tablet_bootstrap.cc:492] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: Bootstrap complete.
I20240503 08:13:14.411387 11822 ts_tablet_manager.cc:1367] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: Time spent bootstrapping tablet: real 2.458s	user 1.140s	sys 0.090s
I20240503 08:13:14.412137 11822 raft_consensus.cc:348] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:14.413144 11822 raft_consensus.cc:729] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456c568b524465bb505956695d13738, State: Initialized, Role: FOLLOWER
I20240503 08:13:14.413390 11822 consensus_queue.cc:260] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 912, Last appended: 1.913, Last appended by leader: 913, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:14.415733 11822 ts_tablet_manager.cc:1398] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:13:14.416764  7744 tablet_copy_service.cc:342] P 359f2f323e79431eac07dc0a3f47a47b: Request end of tablet copy session 1456c568b524465bb505956695d13738-01f87cf2ca7041cd8727e41d994f330c received from {username='slave'} at 127.2.49.133:40821
I20240503 08:13:14.416923  7744 tablet_copy_service.cc:433] P 359f2f323e79431eac07dc0a3f47a47b: ending tablet copy session 1456c568b524465bb505956695d13738-01f87cf2ca7041cd8727e41d994f330c on tablet 01f87cf2ca7041cd8727e41d994f330c with peer 1456c568b524465bb505956695d13738
I20240503 08:13:14.482280  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:13:14.781841  7529 consensus_peers.cc:485] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 -> Peer fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): Couldn't send request to peer fb22180119ec428393402ddbd0b748b6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:14.834511  9058 raft_consensus.cc:1203] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.908->[1.909-1.950]   Dedup: 1.913->[1.914-1.950]
W20240503 08:13:15.101435  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 54749.5us
I20240503 08:13:15.616371 11846 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=780 overwritten=0 applied=780 ignored=0} inserts{seen=1452 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.618963 11846 tablet_bootstrap.cc:492] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: Bootstrap complete.
I20240503 08:13:15.620074 11846 ts_tablet_manager.cc:1367] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: Time spent bootstrapping tablet: real 3.056s	user 0.987s	sys 0.139s
I20240503 08:13:15.621047 11846 raft_consensus.cc:348] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:15.621404 11846 raft_consensus.cc:729] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.621762 11846 consensus_queue.cc:260] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 780, Last appended: 1.780, Last appended by leader: 780, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:15.622484 11846 ts_tablet_manager.cc:1398] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:13:15.627861  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session fb22180119ec428393402ddbd0b748b6-a7f1cb831a9149e494cb31fac27f922d received from {username='slave'} at 127.2.49.132:37923
I20240503 08:13:15.628003  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session fb22180119ec428393402ddbd0b748b6-a7f1cb831a9149e494cb31fac27f922d on tablet a7f1cb831a9149e494cb31fac27f922d with peer fb22180119ec428393402ddbd0b748b6
W20240503 08:13:15.709959  7787 consensus_peers.cc:485] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 -> Peer b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134:37149): Couldn't send request to peer b50f0dfea2284eacbe4184b1f3ae3753. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:15.795857  8918 raft_consensus.cc:1203] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.774->[1.775-1.818]   Dedup: 1.780->[1.781-1.818]
I20240503 08:13:15.910612 11883 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap replayed 1/1 log segments. Stats: ops{read=827 overwritten=0 applied=827 ignored=0} inserts{seen=1510 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:15.912377 11883 tablet_bootstrap.cc:492] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: Bootstrap complete.
I20240503 08:13:15.916658 11883 ts_tablet_manager.cc:1367] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: Time spent bootstrapping tablet: real 2.211s	user 1.043s	sys 0.084s
I20240503 08:13:15.917670 11883 raft_consensus.cc:348] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:15.918036 11883 raft_consensus.cc:729] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50f0dfea2284eacbe4184b1f3ae3753, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.919548 11883 consensus_queue.cc:260] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 827, Last appended: 1.827, Last appended by leader: 827, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:15.928270 11883 ts_tablet_manager.cc:1398] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.003s
I20240503 08:13:15.934947  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-6462c70bafc94b7fb3bcb9502e80e096 received from {username='slave'} at 127.2.49.134:41109
I20240503 08:13:15.935132  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session b50f0dfea2284eacbe4184b1f3ae3753-6462c70bafc94b7fb3bcb9502e80e096 on tablet 6462c70bafc94b7fb3bcb9502e80e096 with peer b50f0dfea2284eacbe4184b1f3ae3753
W20240503 08:13:16.037217  7659 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.129:39639, user_credentials={real_user=slave}} blocked reactor thread for 75561.3us
I20240503 08:13:16.300077  9186 raft_consensus.cc:1203] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.810->[1.811-1.846]   Dedup: 1.827->[1.828-1.846]
I20240503 08:13:16.441045  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.602368  7834 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:16.636149  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:16.648816  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:16.739567  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.785202  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:16.924990  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:13:18.464779  8852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:39615, user_credentials={real_user=slave}} blocked reactor thread for 85001.4us
I20240503 08:13:18.586700  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:18.619570  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:18.707705  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.815405  7834 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:18.840544  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:18.877617  9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.929101  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:19.473881  7852 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6462c70bafc94b7fb3bcb9502e80e096"
dest_uuid: "5d1ee23344854913811909afd87ab724"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33222
I20240503 08:13:19.474151  7852 raft_consensus.cc:595] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Received request to transfer leadership
W20240503 08:13:19.484405 11986 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.494741  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.502697  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.521777  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.530527  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.540160  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.554845  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.566957  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.577886  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.594398  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.611145  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.631170  7813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.649112  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.666431  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.684702  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.705157  7813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.726725  7813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.746397  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.771745  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
W20240503 08:13:19.797580  7813 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
I20240503 08:13:19.818457 11943 raft_consensus.cc:979] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: : Instructing follower b50f0dfea2284eacbe4184b1f3ae3753 to start an election
I20240503 08:13:19.818663 11943 raft_consensus.cc:1067] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Signalling peer b50f0dfea2284eacbe4184b1f3ae3753 to start an election
I20240503 08:13:19.819638  9189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6462c70bafc94b7fb3bcb9502e80e096"
dest_uuid: "b50f0dfea2284eacbe4184b1f3ae3753"
 from {username='slave'} at 127.2.49.131:42257
I20240503 08:13:19.820024  9189 raft_consensus.cc:482] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:19.820230  9189 raft_consensus.cc:3037] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:19.826627  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Service unavailable: leader transfer in progress
I20240503 08:13:19.840685  9189 raft_consensus.cc:504] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:19.843613  9189 leader_election.cc:288] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [CANDIDATE]: Term 2 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273), 1456c568b524465bb505956695d13738 (127.2.49.133:39615)
I20240503 08:13:19.850380  7852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6462c70bafc94b7fb3bcb9502e80e096" candidate_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" candidate_term: 2 candidate_status { last_received { term: 1 index: 889 } } ignore_live_leader: true dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:13:19.850688  7852 raft_consensus.cc:3032] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:19.850812  7852 raft_consensus.cc:729] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:13:19.851163  7852 consensus_queue.cc:260] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 889, Committed index: 889, Last appended: 1.889, Last appended by leader: 889, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:19.851537  7852 raft_consensus.cc:3037] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:19.854221  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Illegal state: replica 5d1ee23344854913811909afd87ab724 is not leader of this config: current role FOLLOWER
I20240503 08:13:19.868655  7852 raft_consensus.cc:2454] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b50f0dfea2284eacbe4184b1f3ae3753 in term 2.
I20240503 08:13:19.870375  9123 leader_election.cc:302] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1ee23344854913811909afd87ab724, b50f0dfea2284eacbe4184b1f3ae3753; no voters: 
W20240503 08:13:19.957026  9123 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.49.131:40273, user_credentials={real_user=slave}} blocked reactor thread for 86842us
I20240503 08:13:19.959107 12136 raft_consensus.cc:2781] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:19.960630  9059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6462c70bafc94b7fb3bcb9502e80e096" candidate_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" candidate_term: 2 candidate_status { last_received { term: 1 index: 889 } } ignore_live_leader: true dest_uuid: "1456c568b524465bb505956695d13738"
I20240503 08:13:19.960884  9059 raft_consensus.cc:3037] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:19.964213 12136 raft_consensus.cc:686] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 LEADER]: Becoming Leader. State: Replica: b50f0dfea2284eacbe4184b1f3ae3753, State: Running, Role: LEADER
I20240503 08:13:19.964973 12136 consensus_queue.cc:237] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 889, Committed index: 889, Last appended: 1.889, Last appended by leader: 889, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:19.972462  7446 catalog_manager.cc:5484] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 reported cstate change: term changed from 1 to 2, leader changed from 5d1ee23344854913811909afd87ab724 (127.2.49.131) to b50f0dfea2284eacbe4184b1f3ae3753 (127.2.49.134). New cstate: current_term: 2 leader_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" committed_config { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:19.977865  9059 raft_consensus.cc:2454] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b50f0dfea2284eacbe4184b1f3ae3753 in term 2.
W20240503 08:13:20.089308  7811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56514: Illegal state: replica 5d1ee23344854913811909afd87ab724 is not leader of this config: current role FOLLOWER
W20240503 08:13:20.167099  9019 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39662: Illegal state: replica 1456c568b524465bb505956695d13738 is not leader of this config: current role FOLLOWER
I20240503 08:13:20.267912  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:20.272760  7852 raft_consensus.cc:1261] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Refusing update from remote peer b50f0dfea2284eacbe4184b1f3ae3753: Log matching property violated. Preceding OpId in replica: term: 1 index: 889. Preceding OpId from leader: term: 2 index: 891. (index mismatch)
I20240503 08:13:20.273530 12143 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 890, Last known committed idx: 889, Time since last communication: 0.000s
I20240503 08:13:20.275344  9057 raft_consensus.cc:1261] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Refusing update from remote peer b50f0dfea2284eacbe4184b1f3ae3753: Log matching property violated. Preceding OpId in replica: term: 1 index: 889. Preceding OpId from leader: term: 2 index: 891. (index mismatch)
I20240503 08:13:20.282208 12143 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 890, Last known committed idx: 889, Time since last communication: 0.000s
I20240503 08:13:20.283408 12074 mvcc.cc:205] Tried to move back new op lower bound from 7023509505109815296 to 7023509503863140352. Current Snapshot: MvccSnapshot[applied={T|T < 7023509505109815296}]
I20240503 08:13:20.290421  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:20.321213  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:20.420907  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:20.499188  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:13:20.499243  7790 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:37243, user_credentials={real_user=slave}} blocked reactor thread for 71931.7us
I20240503 08:13:20.504139  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:13:20.506719  7787 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 93332.8us
I20240503 08:13:20.595449  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:21.201017  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:21.201478  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:21.225612  7702 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:21.423099  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:21.448611  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:21.461032  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:21.540033  8900 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:21.798739  7721 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01f87cf2ca7041cd8727e41d994f330c"
dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:58984
I20240503 08:13:21.799611  7721 raft_consensus.cc:595] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Received request to transfer leadership
W20240503 08:13:21.823936  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.834378  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.840888  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.847672  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.857890  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.868104  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.877105  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.901235  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.925681  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.945318  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.961012  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.977780  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:21.998503  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.026784  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.044932  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.069505  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.092602  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.117904  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.150635  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.177316  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.209595  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.236466  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
W20240503 08:13:22.262178  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
I20240503 08:13:22.276508 12215 raft_consensus.cc:979] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: : Instructing follower fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:13:22.276741 11974 raft_consensus.cc:1067] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Signalling peer fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:13:22.278642  8920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01f87cf2ca7041cd8727e41d994f330c"
dest_uuid: "fb22180119ec428393402ddbd0b748b6"
 from {username='slave'} at 127.2.49.130:51679
W20240503 08:13:22.292621  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
I20240503 08:13:22.311156  8920 raft_consensus.cc:482] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:22.311522  8920 raft_consensus.cc:3037] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:22.320921  7684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Service unavailable: leader transfer in progress
I20240503 08:13:22.324934  8920 raft_consensus.cc:504] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:22.326833  9057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f87cf2ca7041cd8727e41d994f330c" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1075 } } ignore_live_leader: true dest_uuid: "1456c568b524465bb505956695d13738"
I20240503 08:13:22.327181  9057 raft_consensus.cc:3037] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:22.327494  7721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f87cf2ca7041cd8727e41d994f330c" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1075 } } ignore_live_leader: true dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:13:22.327721  7721 raft_consensus.cc:3032] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:22.327816  7721 raft_consensus.cc:729] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Running, Role: LEADER
I20240503 08:13:22.328135  7721 consensus_queue.cc:260] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1075, Committed index: 1075, Last appended: 1.1075, Last appended by leader: 1075, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:22.328533  7721 raft_consensus.cc:3037] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:22.330467  9057 raft_consensus.cc:2454] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 2.
I20240503 08:13:22.333279  8852 leader_election.cc:302] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1456c568b524465bb505956695d13738, fb22180119ec428393402ddbd0b748b6; no voters: 
I20240503 08:13:22.334771 12228 raft_consensus.cc:2781] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:22.338260  7721 raft_consensus.cc:2454] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 2.
I20240503 08:13:22.338863  8920 leader_election.cc:288] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 2 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861), 1456c568b524465bb505956695d13738 (127.2.49.133:39615)
W20240503 08:13:22.349340  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Illegal state: replica 359f2f323e79431eac07dc0a3f47a47b is not leader of this config: current role FOLLOWER
I20240503 08:13:22.378746 12228 raft_consensus.cc:686] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 2 LEADER]: Becoming Leader. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Running, Role: LEADER
I20240503 08:13:22.379228 12228 consensus_queue.cc:237] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1075, Committed index: 1075, Last appended: 1.1075, Last appended by leader: 1075, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:22.382367  7452 catalog_manager.cc:5484] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 reported cstate change: term changed from 1 to 2, leader changed from 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) to fb22180119ec428393402ddbd0b748b6 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 909 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
W20240503 08:13:22.516484  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51142: Illegal state: replica 359f2f323e79431eac07dc0a3f47a47b is not leader of this config: current role FOLLOWER
I20240503 08:13:22.523536  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:22.555285  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:22.621284  9057 raft_consensus.cc:1261] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1075. Preceding OpId from leader: term: 2 index: 1077. (index mismatch)
I20240503 08:13:22.621311  7721 raft_consensus.cc:1261] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1075. Preceding OpId from leader: term: 2 index: 1077. (index mismatch)
I20240503 08:13:22.621976 12228 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1076, Last known committed idx: 1075, Time since last communication: 0.000s
I20240503 08:13:22.636891 12228 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1076, Last known committed idx: 1075, Time since last communication: 0.000s
I20240503 08:13:22.654795  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:22.655465  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:22.655882  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:22.683465  8900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:22.808840  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:13:23.496785  7787 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:37149, user_credentials={real_user=slave}} blocked reactor thread for 56236us
I20240503 08:13:24.245600  7571 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:24.246259  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:24.263085  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:24.267004  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:24.293130  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:24.467706  8900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:24.517925  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:24.813344  7593 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7f1cb831a9149e494cb31fac27f922d"
dest_uuid: "55fc889503854003801521bf97120179"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56218
I20240503 08:13:24.813553  7593 raft_consensus.cc:595] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:24.814152 12307 raft_consensus.cc:979] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: : Instructing follower fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:13:24.814281 12307 raft_consensus.cc:1067] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 LEADER]: Signalling peer fb22180119ec428393402ddbd0b748b6 to start an election
W20240503 08:13:24.819272  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60554: Service unavailable: leader transfer in progress
I20240503 08:13:24.820850  8918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7f1cb831a9149e494cb31fac27f922d"
dest_uuid: "fb22180119ec428393402ddbd0b748b6"
 from {username='slave'} at 127.2.49.129:58271
I20240503 08:13:24.821125  8918 raft_consensus.cc:482] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:24.821261  8918 raft_consensus.cc:3037] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:24.824236  8918 raft_consensus.cc:504] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:24.826452  7720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f1cb831a9149e494cb31fac27f922d" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 974 } } ignore_live_leader: true dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:13:24.826678  7720 raft_consensus.cc:3037] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:24.827853  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60554: Service unavailable: leader transfer in progress
I20240503 08:13:24.830017  7720 raft_consensus.cc:2454] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 2.
I20240503 08:13:24.830971  8854 leader_election.cc:302] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, fb22180119ec428393402ddbd0b748b6; no voters: 
I20240503 08:13:24.831276 12266 raft_consensus.cc:2781] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:24.831374 12266 raft_consensus.cc:686] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 2 LEADER]: Becoming Leader. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Running, Role: LEADER
I20240503 08:13:24.831602 12266 consensus_queue.cc:237] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 974, Committed index: 974, Last appended: 1.974, Last appended by leader: 974, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:24.834352  7446 catalog_manager.cc:5484] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 reported cstate change: term changed from 1 to 2, leader changed from 55fc889503854003801521bf97120179 (127.2.49.129) to fb22180119ec428393402ddbd0b748b6 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
W20240503 08:13:24.838559  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60554: Service unavailable: leader transfer in progress
I20240503 08:13:24.841312  8918 leader_election.cc:288] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 2 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861), 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:13:24.844022  7720 raft_consensus.cc:1226] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Rejecting Update request from peer 55fc889503854003801521bf97120179 for earlier term 1. Current term is 2. Ops: []
I20240503 08:13:24.844127  7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f1cb831a9149e494cb31fac27f922d" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 974 } } ignore_live_leader: true dest_uuid: "55fc889503854003801521bf97120179"
I20240503 08:13:24.844333  7593 raft_consensus.cc:3032] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:24.844398  7593 raft_consensus.cc:729] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 55fc889503854003801521bf97120179, State: Running, Role: LEADER
I20240503 08:13:24.844632  7593 consensus_queue.cc:260] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 974, Committed index: 974, Last appended: 1.974, Last appended by leader: 974, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 775 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:24.844873  7593 raft_consensus.cc:3037] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:24.847618  7593 raft_consensus.cc:2454] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 2.
W20240503 08:13:24.849289  7550 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60554: Illegal state: replica 55fc889503854003801521bf97120179 is not leader of this config: current role FOLLOWER
W20240503 08:13:24.941057  7682 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37374: Illegal state: replica 359f2f323e79431eac07dc0a3f47a47b is not leader of this config: current role FOLLOWER
I20240503 08:13:25.071857  7593 raft_consensus.cc:1261] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 974. Preceding OpId from leader: term: 2 index: 976. (index mismatch)
I20240503 08:13:25.071878  7723 raft_consensus.cc:1261] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 974. Preceding OpId from leader: term: 2 index: 976. (index mismatch)
I20240503 08:13:25.073112 12228 consensus_queue.cc:1035] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 975, Last known committed idx: 973, Time since last communication: 0.000s
I20240503 08:13:25.077618 12228 consensus_queue.cc:1035] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 975, Last known committed idx: 974, Time since last communication: 0.000s
I20240503 08:13:25.992666  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:26.001230  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:26.037427  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:26.170491  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:26.235993  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:26.448314  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:26.483361  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:13:27.336686  7529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 54359us
I20240503 08:13:27.593211  8918 consensus_queue.cc:237] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1014, Committed index: 1014, Last appended: 2.1014, Last appended by leader: 974, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1015 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } }
I20240503 08:13:27.594944  7723 raft_consensus.cc:1261] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1014. Preceding OpId from leader: term: 2 index: 1015. (index mismatch)
I20240503 08:13:27.596419 12228 consensus_queue.cc:1035] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1015, Last known committed idx: 1014, Time since last communication: 0.000s
I20240503 08:13:27.606536 12406 raft_consensus.cc:2932] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 [term 2 LEADER]: Committing config change with OpId 2.1015: config changed from index 775 to 1015, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 1015 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:27.612354  7723 raft_consensus.cc:2932] T a7f1cb831a9149e494cb31fac27f922d P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Committing config change with OpId 2.1015: config changed from index 775 to 1015, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 1015 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:27.625620  7452 catalog_manager.cc:5484] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 reported cstate change: config changed from index 775 to 1015, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 1015 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } }
I20240503 08:13:27.678850  7571 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f1cb831a9149e494cb31fac27f922d with delete_type TABLET_DATA_TOMBSTONED (TS 55fc889503854003801521bf97120179 not found in new config with opid_index 1015) from {username='slave'} at 127.0.0.1:48186
I20240503 08:13:27.679937 12505 tablet_replica.cc:330] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: stopping tablet replica
I20240503 08:13:27.680166 12505 raft_consensus.cc:2229] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:27.680409 12505 raft_consensus.cc:2258] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:27.721848 12505 ts_tablet_manager.cc:1875] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:27.743106 12505 ts_tablet_manager.cc:1888] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1014
I20240503 08:13:27.744143 12505 log.cc:1196] T a7f1cb831a9149e494cb31fac27f922d P 55fc889503854003801521bf97120179: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/a7f1cb831a9149e494cb31fac27f922d
I20240503 08:13:27.745296  7438 catalog_manager.cc:4830] TS 55fc889503854003801521bf97120179 (127.2.49.129:39639): tablet a7f1cb831a9149e494cb31fac27f922d (table rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]) successfully deleted
I20240503 08:13:28.711400  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:28.714475  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:28.750046  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:28.791409  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:28.809979  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:28.816908  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:28.828044  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:29.871237  9186 consensus_queue.cc:237] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1019, Committed index: 1019, Last appended: 2.1019, Last appended by leader: 889, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1020 OBSOLETE_local: false peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:13:29.875950  9058 raft_consensus.cc:1261] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Refusing update from remote peer b50f0dfea2284eacbe4184b1f3ae3753: Log matching property violated. Preceding OpId in replica: term: 2 index: 1019. Preceding OpId from leader: term: 2 index: 1020. (index mismatch)
I20240503 08:13:29.882138 12449 consensus_queue.cc:1035] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1020, Last known committed idx: 1019, Time since last communication: 0.000s
I20240503 08:13:29.891604 12581 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 LEADER]: Committing config change with OpId 2.1020: config changed from index 811 to 1020, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 1020 OBSOLETE_local: false peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:29.894071  9058 raft_consensus.cc:2932] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Committing config change with OpId 2.1020: config changed from index 811 to 1020, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 1020 OBSOLETE_local: false peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:29.901794  7446 catalog_manager.cc:5484] T 6462c70bafc94b7fb3bcb9502e80e096 P 1456c568b524465bb505956695d13738 reported cstate change: config changed from index 811 to 1020, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New cstate: current_term: 2 leader_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" committed_config { opid_index: 1020 OBSOLETE_local: false peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } } }
I20240503 08:13:30.049831  7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 6462c70bafc94b7fb3bcb9502e80e096 with delete_type TABLET_DATA_TOMBSTONED (TS 5d1ee23344854913811909afd87ab724 not found in new config with opid_index 1020) from {username='slave'} at 127.0.0.1:37614
I20240503 08:13:30.055066 12601 tablet_replica.cc:330] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: stopping tablet replica
I20240503 08:13:30.055383 12601 raft_consensus.cc:2229] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.055696 12601 raft_consensus.cc:2258] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.057183 12601 ts_tablet_manager.cc:1875] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.076995 12601 ts_tablet_manager.cc:1888] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1019
I20240503 08:13:30.077231 12601 log.cc:1196] T 6462c70bafc94b7fb3bcb9502e80e096 P 5d1ee23344854913811909afd87ab724: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:30.078996  7437 catalog_manager.cc:4830] TS 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): tablet 6462c70bafc94b7fb3bcb9502e80e096 (table rebalance_test_table_0 [id=9822a155357b4da38dc52ebeab8687aa]) successfully deleted
I20240503 08:13:30.710498  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:30.772886  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:30.852768  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:30.928010  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:31.013249  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:31.062033  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:31.109745  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:31.599239  8920 consensus_queue.cc:237] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1245, Committed index: 1245, Last appended: 2.1245, Last appended by leader: 1075, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1246 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } }
I20240503 08:13:31.601781  9059 raft_consensus.cc:1261] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1245. Preceding OpId from leader: term: 2 index: 1246. (index mismatch)
I20240503 08:13:31.608569 12228 consensus_queue.cc:1035] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1246, Last known committed idx: 1245, Time since last communication: 0.000s
I20240503 08:13:31.638144 12412 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 [term 2 LEADER]: Committing config change with OpId 2.1246: config changed from index 909 to 1246, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 1246 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:31.639278  9059 raft_consensus.cc:2932] T 01f87cf2ca7041cd8727e41d994f330c P 1456c568b524465bb505956695d13738 [term 2 FOLLOWER]: Committing config change with OpId 2.1246: config changed from index 909 to 1246, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 1246 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:31.645990  7446 catalog_manager.cc:5484] T 01f87cf2ca7041cd8727e41d994f330c P fb22180119ec428393402ddbd0b748b6 reported cstate change: config changed from index 909 to 1246, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 1246 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "1456c568b524465bb505956695d13738" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 39615 } } }
I20240503 08:13:31.691620  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f87cf2ca7041cd8727e41d994f330c with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 1246) from {username='slave'} at 127.0.0.1:46506
I20240503 08:13:31.734015 12681 tablet_replica.cc:330] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:13:31.734679 12681 raft_consensus.cc:2229] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:31.735888 12681 raft_consensus.cc:2258] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:31.743197 12681 ts_tablet_manager.cc:1875] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:31.761296 12681 ts_tablet_manager.cc:1888] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1245
I20240503 08:13:31.761582 12681 log.cc:1196] T 01f87cf2ca7041cd8727e41d994f330c P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/01f87cf2ca7041cd8727e41d994f330c
I20240503 08:13:31.763410  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet 01f87cf2ca7041cd8727e41d994f330c (table rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]) successfully deleted
I20240503 08:13:33.135751  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:33.213393  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:33.248132  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:33.288542  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:33.304224  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:33.316700  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:33.405984  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:34.488878  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:34.594053  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:34.636972  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:34.695039  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:34.707134  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:34.730055  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:34.842767  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:13:36.008363  7660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:41615, user_credentials={real_user=slave}} blocked reactor thread for 61647.3us
I20240503 08:13:37.425240  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:37.451052  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:37.456666  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:37.475558  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:37.519559  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:37.552139  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:37.554984  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:38.466742  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:38.500478  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:38.530923  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:38.590005  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:38.856357  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:38.891150  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:38.906754  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:39.900559  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:39.922021  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:39.939749  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:39.945722  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:39.968088  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:39.971464  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:40.013222  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 56ef465a043845eb8a012eb5e7ce8be4 | 127.2.49.190:41099 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                               | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.2.49.148:33259 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 131e5ac1bcce424cbb1f7f60be20a263 | 127.2.49.135:37243 | HEALTHY | <none>   |       0        |       0
 1456c568b524465bb505956695d13738 | 127.2.49.133:39615 | HEALTHY | <none>   |       1        |       0
 359f2f323e79431eac07dc0a3f47a47b | 127.2.49.130:43861 | HEALTHY | <none>   |       3        |       0
 55fc889503854003801521bf97120179 | 127.2.49.129:39639 | HEALTHY | <none>   |       4        |       0
 5d1ee23344854913811909afd87ab724 | 127.2.49.131:40273 | HEALTHY | <none>   |       4        |       0
 b50f0dfea2284eacbe4184b1f3ae3753 | 127.2.49.134:37149 | HEALTHY | <none>   |       1        |       0
 fb22180119ec428393402ddbd0b748b6 | 127.2.49.132:41615 | HEALTHY | <none>   |       3        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.49.129                                                                                                                                                    | experimental          | 127.2.49.129:39639
 local_ip_for_outbound_sockets            | 127.2.49.130                                                                                                                                                    | experimental          | 127.2.49.130:43861
 local_ip_for_outbound_sockets            | 127.2.49.131                                                                                                                                                    | experimental          | 127.2.49.131:40273
 local_ip_for_outbound_sockets            | 127.2.49.132                                                                                                                                                    | experimental          | 127.2.49.132:41615
 local_ip_for_outbound_sockets            | 127.2.49.133                                                                                                                                                    | experimental          | 127.2.49.133:39615
 local_ip_for_outbound_sockets            | 127.2.49.134                                                                                                                                                    | experimental          | 127.2.49.134:37149
 local_ip_for_outbound_sockets            | 127.2.49.135                                                                                                                                                    | experimental          | 127.2.49.135:37243
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                           | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.49.129:39639
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.49.130:43861
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.49.131:40273
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.49.132:41615
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.2.49.133:39615
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.2.49.134:37149
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.2.49.135:37243

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.2.49.148:33259 | all 7 server(s) checked
 time_source         | builtin            | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 08:13:40.083925  2246 log_verifier.cc:126] Checking tablet 01f87cf2ca7041cd8727e41d994f330c
I20240503 08:13:40.582232  2246 log_verifier.cc:177] Verified matching terms for 1378 ops in tablet 01f87cf2ca7041cd8727e41d994f330c
I20240503 08:13:40.583920  2246 log_verifier.cc:126] Checking tablet 0246c726f68848a7ad4620e533f93b53
I20240503 08:13:40.600595  2246 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 0246c726f68848a7ad4620e533f93b53
I20240503 08:13:40.600719  2246 log_verifier.cc:126] Checking tablet 0563aa33bb13450c96f620998b881c61
I20240503 08:13:40.990240  2246 log_verifier.cc:177] Verified matching terms for 1123 ops in tablet 0563aa33bb13450c96f620998b881c61
I20240503 08:13:40.992635  2246 log_verifier.cc:126] Checking tablet 303fded976c342efbccfc375950f78ec
I20240503 08:13:41.457041  2246 log_verifier.cc:177] Verified matching terms for 1269 ops in tablet 303fded976c342efbccfc375950f78ec
I20240503 08:13:41.458477  2246 log_verifier.cc:126] Checking tablet 3242894c953a4f6f8095bfb8c9318855
I20240503 08:13:41.911757  2246 log_verifier.cc:177] Verified matching terms for 1338 ops in tablet 3242894c953a4f6f8095bfb8c9318855
I20240503 08:13:41.913349  2246 log_verifier.cc:126] Checking tablet 44c94e75cff640f8927043b5e071c980
I20240503 08:13:42.317766  2246 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 44c94e75cff640f8927043b5e071c980
I20240503 08:13:42.319123  2246 log_verifier.cc:126] Checking tablet 6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:42.712563  2246 log_verifier.cc:177] Verified matching terms for 1181 ops in tablet 6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:42.713872  2246 log_verifier.cc:126] Checking tablet 65167136cc2d478cb0efd6a374839c48
I20240503 08:13:43.091693  2246 log_verifier.cc:177] Verified matching terms for 1169 ops in tablet 65167136cc2d478cb0efd6a374839c48
I20240503 08:13:43.093094  2246 log_verifier.cc:126] Checking tablet 6c6b0211f6b44c1b92a5be884608d000
I20240503 08:13:43.497334  2246 log_verifier.cc:177] Verified matching terms for 1185 ops in tablet 6c6b0211f6b44c1b92a5be884608d000
I20240503 08:13:43.498735  2246 log_verifier.cc:126] Checking tablet 7e5a34b052e94b7b8a1dbaec2038d921
I20240503 08:13:43.995618  2246 log_verifier.cc:177] Verified matching terms for 1422 ops in tablet 7e5a34b052e94b7b8a1dbaec2038d921
I20240503 08:13:43.997277  2246 log_verifier.cc:126] Checking tablet 80eccc47e060493ebe43b007c7a23f00
I20240503 08:13:44.389575  2246 log_verifier.cc:177] Verified matching terms for 1188 ops in tablet 80eccc47e060493ebe43b007c7a23f00
I20240503 08:13:44.390933  2246 log_verifier.cc:126] Checking tablet 8967d6dac1aa482683c8bad1892dcbde
I20240503 08:13:44.867655  2246 log_verifier.cc:177] Verified matching terms for 1320 ops in tablet 8967d6dac1aa482683c8bad1892dcbde
I20240503 08:13:44.869055  2246 log_verifier.cc:126] Checking tablet 91ae2ad1ed4448689d8fe8eba7942dd9
I20240503 08:13:45.277316  2246 log_verifier.cc:177] Verified matching terms for 1262 ops in tablet 91ae2ad1ed4448689d8fe8eba7942dd9
I20240503 08:13:45.278666  2246 log_verifier.cc:126] Checking tablet 9f5a650ed26a406baa15105ac9fc8f8f
I20240503 08:13:45.724606  2246 log_verifier.cc:177] Verified matching terms for 1204 ops in tablet 9f5a650ed26a406baa15105ac9fc8f8f
I20240503 08:13:45.725905  2246 log_verifier.cc:126] Checking tablet a629b67231f84cdbb5e302265896e25a
I20240503 08:13:46.096767  2246 log_verifier.cc:177] Verified matching terms for 1165 ops in tablet a629b67231f84cdbb5e302265896e25a
I20240503 08:13:46.098114  2246 log_verifier.cc:126] Checking tablet a7f1cb831a9149e494cb31fac27f922d
I20240503 08:13:46.546505  2246 log_verifier.cc:177] Verified matching terms for 1209 ops in tablet a7f1cb831a9149e494cb31fac27f922d
I20240503 08:13:46.557103  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.136:0
--local_ip_for_outbound_sockets=127.2.49.136
--webserver_interface=127.2.49.136
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--builtin_ntp_servers=127.2.49.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:13:46.963358 13038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:46.963672 13038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:46.963795 13038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:46.972222 13038 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:13:46.972358 13038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:46.972530 13038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.136
I20240503 08:13:46.982777 13038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.49.136
--webserver_port=0
--tserver_master_addrs=127.2.49.190:41099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.136
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:46.984256 13038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:46.986330 13038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.003893 13046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.004465 13048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.005087 13045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.005645 13038 server_base.cc:1034] running on GCE node
I20240503 08:13:47.006412 13038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.007556 13038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.008795 13038 hybrid_clock.cc:648] HybridClock initialized: now 1714724027008736 us; error 53 us; skew 500 ppm
I20240503 08:13:47.009141 13038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:47.011831 13038 webserver.cc:460] Webserver started at http://127.2.49.136:39759/ using document root <none> and password file <none>
I20240503 08:13:47.012730 13038 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.012864 13038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.013195 13038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.016001 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "fb1787264a5d40a1a72afabda025bc5e"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-dz87"
I20240503 08:13:47.016894 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "fb1787264a5d40a1a72afabda025bc5e"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-dz87"
I20240503 08:13:47.022773 13038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240503 08:13:47.026791 13054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:13:47.028295 13038 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:13:47.028496 13038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "fb1787264a5d40a1a72afabda025bc5e"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-dz87"
I20240503 08:13:47.028721 13038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.049024 13038 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:13:47.049973 13038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.050300 13038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.051348 13038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:47.053016 13038 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:47.053262 13038 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:13:47.053417 13038 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:47.053505 13038 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:13:47.092005 13038 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.136:43675
I20240503 08:13:47.092079 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.136:43675 every 8 connection(s)
I20240503 08:13:47.093355 13038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:13:47.098085  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 13038
I20240503 08:13:47.098366  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:13:47.111383 13167 heartbeater.cc:347] Connected to a master server at 127.2.49.190:41099
I20240503 08:13:47.111753 13167 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:47.112695 13167 heartbeater.cc:511] Master 127.2.49.190:41099 requested a full tablet report, sending...
I20240503 08:13:47.114332  7452 ts_manager.cc:194] Registered new tserver with Master: fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675)
I20240503 08:13:47.115391  7452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.136:55481
I20240503 08:13:47.716544  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:47.737936  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:47.773671  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:47.791087  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:47.798282  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:47.861511  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:47.901611 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:47.906188  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:48.120034 13167 heartbeater.cc:503] Master 127.2.49.190:41099 was elected leader, sending a full tablet report...
I20240503 08:13:48.154209  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:48.164265  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:48.164712  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:48.165606  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:48.175810  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:48.187438  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:48.230576 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:48.232630  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:48.335618  7723 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1188, Committed index: 1188, Last appended: 2.1188, Last appended by leader: 645, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:48.338075  8918 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1188. Preceding OpId from leader: term: 2 index: 1189. (index mismatch)
I20240503 08:13:48.338706 13275 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1189, Last known committed idx: 1188, Time since last communication: 0.000s
I20240503 08:13:48.341194 12849 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Committing config change with OpId 2.1189: config changed from index 678 to 1189, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.341804  8918 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Committing config change with OpId 2.1189: config changed from index 678 to 1189, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
W20240503 08:13:48.344254  7657 consensus_peers.cc:485] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80eccc47e060493ebe43b007c7a23f00. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.347191  7444 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index 678 to 1189, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New cstate: current_term: 2 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.381618  7723 consensus_queue.cc:237] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1338, Committed index: 1338, Last appended: 1.1338, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1339 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:48.383838  7593 raft_consensus.cc:1261] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1338. Preceding OpId from leader: term: 1 index: 1339. (index mismatch)
W20240503 08:13:48.384176  7657 consensus_peers.cc:485] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3242894c953a4f6f8095bfb8c9318855. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.384483 13275 consensus_queue.cc:1035] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1339, Last known committed idx: 1338, Time since last communication: 0.000s
I20240503 08:13:48.387094 12849 raft_consensus.cc:2932] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.1339: config changed from index -1 to 1339, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1339 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.387822  7593 raft_consensus.cc:2932] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Committing config change with OpId 1.1339: config changed from index -1 to 1339, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1339 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.393384  7452 catalog_manager.cc:5484] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index -1 to 1339, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 1339 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.423617  7851 consensus_queue.cc:237] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1269, Committed index: 1269, Last appended: 1.1269, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:48.426110  9314 raft_consensus.cc:1261] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Refusing update from remote peer 5d1ee23344854913811909afd87ab724: Log matching property violated. Preceding OpId in replica: term: 1 index: 1269. Preceding OpId from leader: term: 1 index: 1270. (index mismatch)
I20240503 08:13:48.426724 13026 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1270, Last known committed idx: 1269, Time since last communication: 0.000s
I20240503 08:13:48.429526 13315 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Committing config change with OpId 1.1270: config changed from index 324 to 1270, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.430351  9314 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Committing config change with OpId 1.1270: config changed from index 324 to 1270, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
W20240503 08:13:48.431846  7787 consensus_peers.cc:485] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 303fded976c342efbccfc375950f78ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.437438  7444 catalog_manager.cc:5484] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 reported cstate change: config changed from index 324 to 1270, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "5d1ee23344854913811909afd87ab724" committed_config { opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.468472  7593 consensus_queue.cc:237] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1169, Committed index: 1169, Last appended: 1.1169, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1170 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:48.471961  8918 raft_consensus.cc:1261] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 1169. Preceding OpId from leader: term: 1 index: 1170. (index mismatch)
I20240503 08:13:48.472527 12846 consensus_queue.cc:1035] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1170, Last known committed idx: 1169, Time since last communication: 0.000s
I20240503 08:13:48.476011 12846 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.1170: config changed from index 218 to 1170, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1170 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.476482  8918 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Committing config change with OpId 1.1170: config changed from index 218 to 1170, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New config: { opid_index: 1170 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:48.481704  7452 catalog_manager.cc:5484] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 reported cstate change: config changed from index 218 to 1170, VOTER fb1787264a5d40a1a72afabda025bc5e (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 1170 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
W20240503 08:13:48.493633  7526 consensus_peers.cc:485] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65167136cc2d478cb0efd6a374839c48. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:48.601899  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:48.602741  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:48.617024  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:48.621407  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:48.646142  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:48.651867 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:48.669413  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:48.673399  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:48.758021 13379 ts_tablet_manager.cc:902] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Initiating tablet copy from peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:13:48.761209 13379 tablet_copy_client.cc:308] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:43861
I20240503 08:13:48.762257 13389 ts_tablet_manager.cc:902] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: Initiating tablet copy from peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
I20240503 08:13:48.779201 13389 tablet_copy_client.cc:308] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:43861
I20240503 08:13:48.789397  7744 tablet_copy_service.cc:140] P 359f2f323e79431eac07dc0a3f47a47b: Received BeginTabletCopySession request for tablet 80eccc47e060493ebe43b007c7a23f00 from peer fb1787264a5d40a1a72afabda025bc5e ({username='slave'} at 127.2.49.136:43419)
I20240503 08:13:48.789599  7744 tablet_copy_service.cc:161] P 359f2f323e79431eac07dc0a3f47a47b: Beginning new tablet copy session on tablet 80eccc47e060493ebe43b007c7a23f00 from peer fb1787264a5d40a1a72afabda025bc5e at {username='slave'} at 127.2.49.136:43419: session id = fb1787264a5d40a1a72afabda025bc5e-80eccc47e060493ebe43b007c7a23f00
I20240503 08:13:48.792717  7744 tablet_copy_source_session.cc:205] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:48.793313  7743 tablet_copy_service.cc:140] P 359f2f323e79431eac07dc0a3f47a47b: Received BeginTabletCopySession request for tablet 3242894c953a4f6f8095bfb8c9318855 from peer fb1787264a5d40a1a72afabda025bc5e ({username='slave'} at 127.2.49.136:43419)
I20240503 08:13:48.793484  7743 tablet_copy_service.cc:161] P 359f2f323e79431eac07dc0a3f47a47b: Beginning new tablet copy session on tablet 3242894c953a4f6f8095bfb8c9318855 from peer fb1787264a5d40a1a72afabda025bc5e at {username='slave'} at 127.2.49.136:43419: session id = fb1787264a5d40a1a72afabda025bc5e-3242894c953a4f6f8095bfb8c9318855
I20240503 08:13:48.797953  7743 tablet_copy_source_session.cc:205] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:48.798749 13379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eccc47e060493ebe43b007c7a23f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.799507 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3242894c953a4f6f8095bfb8c9318855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.813655 13379 tablet_copy_client.cc:783] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:48.813977 13389 tablet_copy_client.cc:783] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:48.814379 13379 tablet_copy_client.cc:647] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:48.814438 13389 tablet_copy_client.cc:647] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:48.821424 13389 tablet_copy_client.cc:523] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:48.823139 13379 tablet_copy_client.cc:523] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:48.827298 13379 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap starting.
I20240503 08:13:48.828002 13389 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap starting.
I20240503 08:13:48.943377  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:48.943630  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:48.947739  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:48.959134  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:48.979295  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:49.009305  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:49.010418 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:49.011940  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:49.084677 13420 ts_tablet_manager.cc:902] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: Initiating tablet copy from peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:13:49.096899 13422 ts_tablet_manager.cc:902] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: Initiating tablet copy from peer 55fc889503854003801521bf97120179 (127.2.49.129:39639)
I20240503 08:13:49.114390 13420 tablet_copy_client.cc:308] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:40273
I20240503 08:13:49.123531  7874 tablet_copy_service.cc:140] P 5d1ee23344854913811909afd87ab724: Received BeginTabletCopySession request for tablet 303fded976c342efbccfc375950f78ec from peer fb1787264a5d40a1a72afabda025bc5e ({username='slave'} at 127.2.49.136:46215)
I20240503 08:13:49.123718  7874 tablet_copy_service.cc:161] P 5d1ee23344854913811909afd87ab724: Beginning new tablet copy session on tablet 303fded976c342efbccfc375950f78ec from peer fb1787264a5d40a1a72afabda025bc5e at {username='slave'} at 127.2.49.136:46215: session id = fb1787264a5d40a1a72afabda025bc5e-303fded976c342efbccfc375950f78ec
I20240503 08:13:49.127338  7874 tablet_copy_source_session.cc:205] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.138347 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303fded976c342efbccfc375950f78ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.168143 13420 tablet_copy_client.cc:783] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.168627 13420 tablet_copy_client.cc:647] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.181411 13420 tablet_copy_client.cc:523] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.185671 13420 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: Bootstrap starting.
I20240503 08:13:49.209900 13422 tablet_copy_client.cc:308] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:39639
I20240503 08:13:49.222834  7613 tablet_copy_service.cc:140] P 55fc889503854003801521bf97120179: Received BeginTabletCopySession request for tablet 65167136cc2d478cb0efd6a374839c48 from peer fb1787264a5d40a1a72afabda025bc5e ({username='slave'} at 127.2.49.136:37983)
I20240503 08:13:49.223042  7613 tablet_copy_service.cc:161] P 55fc889503854003801521bf97120179: Beginning new tablet copy session on tablet 65167136cc2d478cb0efd6a374839c48 from peer fb1787264a5d40a1a72afabda025bc5e at {username='slave'} at 127.2.49.136:37983: session id = fb1787264a5d40a1a72afabda025bc5e-65167136cc2d478cb0efd6a374839c48
I20240503 08:13:49.226052  7613 tablet_copy_source_session.cc:205] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:49.227947 13422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65167136cc2d478cb0efd6a374839c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.232486 13379 log.cc:824] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:49.236765 13422 tablet_copy_client.cc:783] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:49.237087 13422 tablet_copy_client.cc:647] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:49.250283 13422 tablet_copy_client.cc:523] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:49.256003 13422 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap starting.
I20240503 08:13:49.708108  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:49.714857  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:49.734270  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:49.751026  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:49.773069  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:49.789803  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:49.835920  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:49.887168 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:50.415935  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:50.419806  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:50.424546  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:50.436829  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:50.456996  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:50.535341 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:50.543285  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:50.549214  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:51.112520  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:13:51.119119  7657 consensus_peers.cc:485] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:51.172821  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:51.236106  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:51.258193  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:51.264019  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:51.277714  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:51.295993 13379 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1189 overwritten=0 applied=1189 ignored=0} inserts{seen=1858 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:51.296761 13379 tablet_bootstrap.cc:492] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap complete.
I20240503 08:13:51.297511 13379 ts_tablet_manager.cc:1367] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Time spent bootstrapping tablet: real 2.470s	user 1.422s	sys 0.108s
I20240503 08:13:51.302971  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:51.303489 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:51.306529 13379 raft_consensus.cc:348] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:51.307251 13379 raft_consensus.cc:729] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1787264a5d40a1a72afabda025bc5e, State: Initialized, Role: FOLLOWER
I20240503 08:13:51.308120 13379 consensus_queue.cc:260] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1189, Last appended: 2.1189, Last appended by leader: 1189, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:51.330598 13379 ts_tablet_manager.cc:1398] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e: Time spent starting tablet: real 0.033s	user 0.018s	sys 0.000s
I20240503 08:13:51.332495  7744 tablet_copy_service.cc:342] P 359f2f323e79431eac07dc0a3f47a47b: Request end of tablet copy session fb1787264a5d40a1a72afabda025bc5e-80eccc47e060493ebe43b007c7a23f00 received from {username='slave'} at 127.2.49.136:43419
I20240503 08:13:51.332635  7744 tablet_copy_service.cc:433] P 359f2f323e79431eac07dc0a3f47a47b: ending tablet copy session fb1787264a5d40a1a72afabda025bc5e-80eccc47e060493ebe43b007c7a23f00 on tablet 80eccc47e060493ebe43b007c7a23f00 with peer fb1787264a5d40a1a72afabda025bc5e
W20240503 08:13:51.393342  7657 consensus_peers.cc:485] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:51.599944 13121 raft_consensus.cc:1203] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1188->[2.1189-2.1189]   Dedup: 2.1189->[]
W20240503 08:13:51.662065  7526 consensus_peers.cc:485] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:51.727362  7787 consensus_peers.cc:485] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 -> Peer fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675): Couldn't send request to peer fb1787264a5d40a1a72afabda025bc5e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:51.896142  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:51.911981  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:51.917469  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:51.936693  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:51.966677 13422 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1170 overwritten=0 applied=1170 ignored=0} inserts{seen=1840 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:51.967288 13422 tablet_bootstrap.cc:492] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap complete.
I20240503 08:13:51.967742 13422 ts_tablet_manager.cc:1367] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: Time spent bootstrapping tablet: real 2.712s	user 1.320s	sys 0.135s
I20240503 08:13:51.969493 13422 raft_consensus.cc:348] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1170 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:51.970235 13422 raft_consensus.cc:729] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1787264a5d40a1a72afabda025bc5e, State: Initialized, Role: FOLLOWER
I20240503 08:13:51.970703 13422 consensus_queue.cc:260] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1170, Last appended: 1.1170, Last appended by leader: 1170, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1170 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:51.971544 13422 ts_tablet_manager.cc:1398] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 08:13:51.972326  7613 tablet_copy_service.cc:342] P 55fc889503854003801521bf97120179: Request end of tablet copy session fb1787264a5d40a1a72afabda025bc5e-65167136cc2d478cb0efd6a374839c48 received from {username='slave'} at 127.2.49.136:37983
I20240503 08:13:51.972467  7613 tablet_copy_service.cc:433] P 55fc889503854003801521bf97120179: ending tablet copy session fb1787264a5d40a1a72afabda025bc5e-65167136cc2d478cb0efd6a374839c48 on tablet 65167136cc2d478cb0efd6a374839c48 with peer fb1787264a5d40a1a72afabda025bc5e
I20240503 08:13:51.991925  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:52.014972 13389 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1339 overwritten=0 applied=1339 ignored=0} inserts{seen=2011 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:52.015707 13389 tablet_bootstrap.cc:492] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: Bootstrap complete.
I20240503 08:13:52.023391  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:52.024132 13389 ts_tablet_manager.cc:1367] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: Time spent bootstrapping tablet: real 3.196s	user 1.545s	sys 0.163s
I20240503 08:13:52.027276 13389 raft_consensus.cc:348] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1339 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:52.033792 13389 raft_consensus.cc:729] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1787264a5d40a1a72afabda025bc5e, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.038350 13389 consensus_queue.cc:260] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1339, Last appended: 1.1339, Last appended by leader: 1339, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1339 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:52.039526 13121 raft_consensus.cc:1203] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1169->[1.1170-1.1170]   Dedup: 1.1170->[]
I20240503 08:13:52.051023 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:52.055042 13389 ts_tablet_manager.cc:1398] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e: Time spent starting tablet: real 0.030s	user 0.004s	sys 0.000s
I20240503 08:13:52.061250  7744 tablet_copy_service.cc:342] P 359f2f323e79431eac07dc0a3f47a47b: Request end of tablet copy session fb1787264a5d40a1a72afabda025bc5e-3242894c953a4f6f8095bfb8c9318855 received from {username='slave'} at 127.2.49.136:43419
I20240503 08:13:52.061391  7744 tablet_copy_service.cc:433] P 359f2f323e79431eac07dc0a3f47a47b: ending tablet copy session fb1787264a5d40a1a72afabda025bc5e-3242894c953a4f6f8095bfb8c9318855 on tablet 3242894c953a4f6f8095bfb8c9318855 with peer fb1787264a5d40a1a72afabda025bc5e
I20240503 08:13:52.076908  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:52.170495 13420 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1270 overwritten=0 applied=1270 ignored=0} inserts{seen=1953 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:52.170992 13420 tablet_bootstrap.cc:492] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: Bootstrap complete.
I20240503 08:13:52.171409 13420 ts_tablet_manager.cc:1367] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: Time spent bootstrapping tablet: real 2.986s	user 1.445s	sys 0.159s
I20240503 08:13:52.172082 13420 raft_consensus.cc:348] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:52.172258 13420 raft_consensus.cc:729] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1787264a5d40a1a72afabda025bc5e, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.172468 13420 consensus_queue.cc:260] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1270, Last appended: 1.1270, Last appended by leader: 1270, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:52.172945 13420 ts_tablet_manager.cc:1398] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:13:52.173763  7874 tablet_copy_service.cc:342] P 5d1ee23344854913811909afd87ab724: Request end of tablet copy session fb1787264a5d40a1a72afabda025bc5e-303fded976c342efbccfc375950f78ec received from {username='slave'} at 127.2.49.136:46215
I20240503 08:13:52.173892  7874 tablet_copy_service.cc:433] P 5d1ee23344854913811909afd87ab724: ending tablet copy session fb1787264a5d40a1a72afabda025bc5e-303fded976c342efbccfc375950f78ec on tablet 303fded976c342efbccfc375950f78ec with peer fb1787264a5d40a1a72afabda025bc5e
I20240503 08:13:52.318770 13121 raft_consensus.cc:1203] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1338->[1.1339-1.1339]   Dedup: 1.1339->[]
I20240503 08:13:52.488348  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:52.491055  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:52.502118  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:52.502861  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:52.527644 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:52.531664  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:52.543653  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:52.568245  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:52.591056 13121 raft_consensus.cc:1203] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1269->[1.1270-1.1270]   Dedup: 1.1270->[]
I20240503 08:13:52.745173  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:52.759096  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:52.770859  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:52.773548  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:52.832343  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:52.845647  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:52.865792  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:52.884996 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.038750  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.043368  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.055554  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.064486  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.066865  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:53.101234  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.112958  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:53.136586 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.216467  7723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00"
dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:48292
I20240503 08:13:53.216696  7723 raft_consensus.cc:595] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Received request to transfer leadership
I20240503 08:13:53.313776  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.316715  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.340576  7702 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:53.361572  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.365720  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.396761  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.412541 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.426034  8900 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:53.463992 13766 raft_consensus.cc:979] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: : Instructing follower fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:13:53.464254 12849 raft_consensus.cc:1067] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Signalling peer fb22180119ec428393402ddbd0b748b6 to start an election
I20240503 08:13:53.464933  8918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00"
dest_uuid: "fb22180119ec428393402ddbd0b748b6"
 from {username='slave'} at 127.2.49.130:51679
I20240503 08:13:53.465135  8918 raft_consensus.cc:482] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:53.465257  8918 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:53.467357  8918 raft_consensus.cc:504] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:53.474853  8918 leader_election.cc:288] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 3 election: Requested vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861), fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675)
I20240503 08:13:53.475356  7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1189 } } ignore_live_leader: true dest_uuid: "359f2f323e79431eac07dc0a3f47a47b"
I20240503 08:13:53.475606  7723 raft_consensus.cc:3032] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:53.475683  7723 raft_consensus.cc:729] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 359f2f323e79431eac07dc0a3f47a47b, State: Running, Role: LEADER
I20240503 08:13:53.475977  7723 consensus_queue.cc:260] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1189, Committed index: 1189, Last appended: 2.1189, Last appended by leader: 1189, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:53.476414  7723 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:53.479547  7723 raft_consensus.cc:2454] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 3.
I20240503 08:13:53.480173  8854 leader_election.cc:302] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f2f323e79431eac07dc0a3f47a47b, fb22180119ec428393402ddbd0b748b6; no voters: 
I20240503 08:13:53.480530 12853 raft_consensus.cc:2781] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:53.481069 12853 raft_consensus.cc:686] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 3 LEADER]: Becoming Leader. State: Replica: fb22180119ec428393402ddbd0b748b6, State: Running, Role: LEADER
I20240503 08:13:53.481436 12853 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1189, Committed index: 1189, Last appended: 2.1189, Last appended by leader: 1189, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:53.485978  7447 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 reported cstate change: term changed from 2 to 3, leader changed from 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) to fb22180119ec428393402ddbd0b748b6 (127.2.49.132). New cstate: current_term: 3 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 1189 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:53.488126 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eccc47e060493ebe43b007c7a23f00" candidate_uuid: "fb22180119ec428393402ddbd0b748b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1189 } } ignore_live_leader: true dest_uuid: "fb1787264a5d40a1a72afabda025bc5e"
I20240503 08:13:53.488437 13121 raft_consensus.cc:3037] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:53.492580 13121 raft_consensus.cc:2454] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb22180119ec428393402ddbd0b748b6 in term 3.
I20240503 08:13:53.640974  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.652248  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:53.652750  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.656376  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.686452  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.687733  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.702653  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.702773 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.822554  7593 consensus_queue.cc:237] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1170, Committed index: 1170, Last appended: 1.1170, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1171 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:53.824584 13121 raft_consensus.cc:1261] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Refusing update from remote peer 55fc889503854003801521bf97120179: Log matching property violated. Preceding OpId in replica: term: 1 index: 1170. Preceding OpId from leader: term: 1 index: 1171. (index mismatch)
I20240503 08:13:53.825218 12846 consensus_queue.cc:1035] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1171, Last known committed idx: 1170, Time since last communication: 0.000s
I20240503 08:13:53.829200 13842 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 [term 1 LEADER]: Committing config change with OpId 1.1171: config changed from index 1170 to 1171, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) evicted. New config: { opid_index: 1171 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:53.829994 13121 raft_consensus.cc:2932] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Committing config change with OpId 1.1171: config changed from index 1170 to 1171, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) evicted. New config: { opid_index: 1171 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:53.833989  7447 catalog_manager.cc:5484] T 65167136cc2d478cb0efd6a374839c48 P 55fc889503854003801521bf97120179 reported cstate change: config changed from index 1170 to 1171, VOTER fb22180119ec428393402ddbd0b748b6 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "55fc889503854003801521bf97120179" committed_config { opid_index: 1171 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:53.844166  8900 tablet_service.cc:1514] Processing DeleteTablet for tablet 65167136cc2d478cb0efd6a374839c48 with delete_type TABLET_DATA_TOMBSTONED (TS fb22180119ec428393402ddbd0b748b6 not found in new config with opid_index 1171) from {username='slave'} at 127.0.0.1:34206
I20240503 08:13:53.845082 13850 tablet_replica.cc:330] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: stopping tablet replica
I20240503 08:13:53.845544 13850 raft_consensus.cc:2229] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:53.845932 13850 raft_consensus.cc:2258] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:53.848470 13850 ts_tablet_manager.cc:1875] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:53.855451 13850 ts_tablet_manager.cc:1888] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1170
I20240503 08:13:53.855618 13850 log.cc:1196] T 65167136cc2d478cb0efd6a374839c48 P fb22180119ec428393402ddbd0b748b6: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/65167136cc2d478cb0efd6a374839c48
I20240503 08:13:53.856714  7439 catalog_manager.cc:4830] TS fb22180119ec428393402ddbd0b748b6 (127.2.49.132:41615): tablet 65167136cc2d478cb0efd6a374839c48 (table rebalance_test_table_4 [id=cb95e97e3d554c499c162fbea5b8f2f5]) successfully deleted
I20240503 08:13:53.897300 13121 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 3 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1189. Preceding OpId from leader: term: 3 index: 1190. (index mismatch)
I20240503 08:13:53.898047 12853 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1190, Last known committed idx: 1189, Time since last communication: 0.000s
I20240503 08:13:53.904042  7723 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 3 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1189. Preceding OpId from leader: term: 3 index: 1190. (index mismatch)
I20240503 08:13:53.904708 12853 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1190, Last known committed idx: 1189, Time since last communication: 0.000s
I20240503 08:13:53.945041  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:53.951722  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:53.954995  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.970234  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:53.984337  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:53.985234  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:54.003219 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:54.057984  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.166874  7723 consensus_queue.cc:237] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1339, Committed index: 1339, Last appended: 1.1339, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1340 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:54.168848 13121 raft_consensus.cc:1261] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Refusing update from remote peer 359f2f323e79431eac07dc0a3f47a47b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1339. Preceding OpId from leader: term: 1 index: 1340. (index mismatch)
I20240503 08:13:54.169430 13766 consensus_queue.cc:1035] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1340, Last known committed idx: 1339, Time since last communication: 0.000s
I20240503 08:13:54.172780 13905 raft_consensus.cc:2932] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b [term 1 LEADER]: Committing config change with OpId 1.1340: config changed from index 1339 to 1340, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 1340 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:54.173452 13121 raft_consensus.cc:2932] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Committing config change with OpId 1.1340: config changed from index 1339 to 1340, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New config: { opid_index: 1340 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:54.177242  7443 catalog_manager.cc:5484] T 3242894c953a4f6f8095bfb8c9318855 P 359f2f323e79431eac07dc0a3f47a47b reported cstate change: config changed from index 1339 to 1340, VOTER 55fc889503854003801521bf97120179 (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "359f2f323e79431eac07dc0a3f47a47b" committed_config { opid_index: 1340 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:54.181072  7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 3242894c953a4f6f8095bfb8c9318855 with delete_type TABLET_DATA_TOMBSTONED (TS 55fc889503854003801521bf97120179 not found in new config with opid_index 1340) from {username='slave'} at 127.0.0.1:48186
I20240503 08:13:54.181728 13906 tablet_replica.cc:330] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: stopping tablet replica
I20240503 08:13:54.181905 13906 raft_consensus.cc:2229] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:54.182196 13906 raft_consensus.cc:2258] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:54.183559 13906 ts_tablet_manager.cc:1875] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:54.188469 13906 ts_tablet_manager.cc:1888] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1339
I20240503 08:13:54.188607 13906 log.cc:1196] T 3242894c953a4f6f8095bfb8c9318855 P 55fc889503854003801521bf97120179: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/3242894c953a4f6f8095bfb8c9318855
I20240503 08:13:54.189545  7438 catalog_manager.cc:4830] TS 55fc889503854003801521bf97120179 (127.2.49.129:39639): tablet 3242894c953a4f6f8095bfb8c9318855 (table rebalance_test_table_2 [id=1610d4e6b14e452080592c64036ae9e0]) successfully deleted
I20240503 08:13:54.482456  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:54.490810  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:54.505827  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.508031  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:54.534256  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:54.547317  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.567183  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.573490 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:54.655334  7851 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "303fded976c342efbccfc375950f78ec"
dest_uuid: "5d1ee23344854913811909afd87ab724"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45978
I20240503 08:13:54.655546  7851 raft_consensus.cc:595] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:54.747469  9297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:54.750787  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:54.752933  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:54.777141  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.791738  7834 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.817893  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:54.858435 13988 raft_consensus.cc:979] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: : Instructing follower 131e5ac1bcce424cbb1f7f60be20a263 to start an election
I20240503 08:13:54.862792 13988 raft_consensus.cc:1067] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Signalling peer 131e5ac1bcce424cbb1f7f60be20a263 to start an election
I20240503 08:13:54.864634  9314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "303fded976c342efbccfc375950f78ec"
dest_uuid: "131e5ac1bcce424cbb1f7f60be20a263"
 from {username='slave'} at 127.2.49.131:38325
I20240503 08:13:54.865010  9314 raft_consensus.cc:482] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:54.865231  9314 raft_consensus.cc:3037] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:54.868594  9314 raft_consensus.cc:504] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:54.869436 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:54.884358  9314 leader_election.cc:288] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 election: Requested vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273), fb1787264a5d40a1a72afabda025bc5e (127.2.49.136:43675)
I20240503 08:13:54.905061  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:54.919064  7851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303fded976c342efbccfc375950f78ec" candidate_uuid: "131e5ac1bcce424cbb1f7f60be20a263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1270 } } ignore_live_leader: true dest_uuid: "5d1ee23344854913811909afd87ab724"
I20240503 08:13:54.919458  7851 raft_consensus.cc:3032] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:54.919598  7851 raft_consensus.cc:729] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d1ee23344854913811909afd87ab724, State: Running, Role: LEADER
I20240503 08:13:54.920076  7851 consensus_queue.cc:260] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1270, Committed index: 1270, Last appended: 1.1270, Last appended by leader: 1270, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:54.920671  7851 raft_consensus.cc:3037] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:54.925889  7851 raft_consensus.cc:2454] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131e5ac1bcce424cbb1f7f60be20a263 in term 2.
I20240503 08:13:54.926823  9253 leader_election.cc:302] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 131e5ac1bcce424cbb1f7f60be20a263, 5d1ee23344854913811909afd87ab724; no voters: 
I20240503 08:13:54.934744 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303fded976c342efbccfc375950f78ec" candidate_uuid: "131e5ac1bcce424cbb1f7f60be20a263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1270 } } ignore_live_leader: true dest_uuid: "fb1787264a5d40a1a72afabda025bc5e"
I20240503 08:13:54.935029 13121 raft_consensus.cc:3037] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:54.937999 13121 raft_consensus.cc:2454] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131e5ac1bcce424cbb1f7f60be20a263 in term 2.
I20240503 08:13:54.953775 13994 raft_consensus.cc:2781] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:54.958317 13994 raft_consensus.cc:686] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 2 LEADER]: Becoming Leader. State: Replica: 131e5ac1bcce424cbb1f7f60be20a263, State: Running, Role: LEADER
I20240503 08:13:54.958925 13994 consensus_queue.cc:237] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1270, Committed index: 1270, Last appended: 1.1270, Last appended by leader: 1270, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:54.965531  7446 catalog_manager.cc:5484] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 reported cstate change: term changed from 1 to 2, leader changed from 5d1ee23344854913811909afd87ab724 (127.2.49.131) to 131e5ac1bcce424cbb1f7f60be20a263 (127.2.49.135). New cstate: current_term: 2 leader_uuid: "131e5ac1bcce424cbb1f7f60be20a263" committed_config { opid_index: 1270 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.151960  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.156360  9297 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.160730  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:55.173736  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:55.176342  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:55.177632  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.188012 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:55.214020  7834 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:55.304644  8918 consensus_queue.cc:237] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1190, Committed index: 1190, Last appended: 3.1190, Last appended by leader: 1189, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1191 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:55.306675 13121 raft_consensus.cc:1261] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 3 FOLLOWER]: Refusing update from remote peer fb22180119ec428393402ddbd0b748b6: Log matching property violated. Preceding OpId in replica: term: 3 index: 1190. Preceding OpId from leader: term: 3 index: 1191. (index mismatch)
I20240503 08:13:55.307291 12853 consensus_queue.cc:1035] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1191, Last known committed idx: 1190, Time since last communication: 0.000s
I20240503 08:13:55.310279 14064 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 [term 3 LEADER]: Committing config change with OpId 3.1191: config changed from index 1189 to 1191, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 1191 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.310926 13121 raft_consensus.cc:2932] T 80eccc47e060493ebe43b007c7a23f00 P fb1787264a5d40a1a72afabda025bc5e [term 3 FOLLOWER]: Committing config change with OpId 3.1191: config changed from index 1189 to 1191, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New config: { opid_index: 1191 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.317263  7444 catalog_manager.cc:5484] T 80eccc47e060493ebe43b007c7a23f00 P fb22180119ec428393402ddbd0b748b6 reported cstate change: config changed from index 1189 to 1191, VOTER 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130) evicted. New cstate: current_term: 3 leader_uuid: "fb22180119ec428393402ddbd0b748b6" committed_config { opid_index: 1191 OBSOLETE_local: false peers { permanent_uuid: "fb22180119ec428393402ddbd0b748b6" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 41615 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.321818  7702 tablet_service.cc:1514] Processing DeleteTablet for tablet 80eccc47e060493ebe43b007c7a23f00 with delete_type TABLET_DATA_TOMBSTONED (TS 359f2f323e79431eac07dc0a3f47a47b not found in new config with opid_index 1191) from {username='slave'} at 127.0.0.1:46506
I20240503 08:13:55.324239 14068 tablet_replica.cc:330] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: stopping tablet replica
I20240503 08:13:55.324429 14068 raft_consensus.cc:2229] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:55.324587 14068 raft_consensus.cc:2258] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:55.326300 14068 ts_tablet_manager.cc:1875] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:55.330535 14068 ts_tablet_manager.cc:1888] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1190
I20240503 08:13:55.330673 14068 log.cc:1196] T 80eccc47e060493ebe43b007c7a23f00 P 359f2f323e79431eac07dc0a3f47a47b: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/80eccc47e060493ebe43b007c7a23f00
I20240503 08:13:55.331645  7439 catalog_manager.cc:4830] TS 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): tablet 80eccc47e060493ebe43b007c7a23f00 (table rebalance_test_table_3 [id=2bb106e82ba64c3b8ac8cc46227b30a9]) successfully deleted
I20240503 08:13:55.417174  7851 raft_consensus.cc:1261] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Refusing update from remote peer 131e5ac1bcce424cbb1f7f60be20a263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1270. Preceding OpId from leader: term: 2 index: 1271. (index mismatch)
I20240503 08:13:55.418076 13994 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1271, Last known committed idx: 1270, Time since last communication: 0.000s
I20240503 08:13:55.434725 13121 raft_consensus.cc:1261] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Refusing update from remote peer 131e5ac1bcce424cbb1f7f60be20a263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1270. Preceding OpId from leader: term: 2 index: 1271. (index mismatch)
I20240503 08:13:55.435705 13994 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1271, Last known committed idx: 1270, Time since last communication: 0.000s
I20240503 08:13:55.482607  9297 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.509348  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:55.528430  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:55.541011  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.546630 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:55.559964  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.564263  7834 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:55.605036  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:55.705636  9314 consensus_queue.cc:237] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1271, Committed index: 1271, Last appended: 2.1271, Last appended by leader: 1270, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1272 OBSOLETE_local: false peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:13:55.707238 13121 raft_consensus.cc:1261] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Refusing update from remote peer 131e5ac1bcce424cbb1f7f60be20a263: Log matching property violated. Preceding OpId in replica: term: 2 index: 1271. Preceding OpId from leader: term: 2 index: 1272. (index mismatch)
I20240503 08:13:55.707779 13994 consensus_queue.cc:1035] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1272, Last known committed idx: 1271, Time since last communication: 0.000s
I20240503 08:13:55.709952 13995 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 [term 2 LEADER]: Committing config change with OpId 2.1272: config changed from index 1270 to 1272, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 1272 OBSOLETE_local: false peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.710552 13121 raft_consensus.cc:2932] T 303fded976c342efbccfc375950f78ec P fb1787264a5d40a1a72afabda025bc5e [term 2 FOLLOWER]: Committing config change with OpId 2.1272: config changed from index 1270 to 1272, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New config: { opid_index: 1272 OBSOLETE_local: false peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.714646  7446 catalog_manager.cc:5484] T 303fded976c342efbccfc375950f78ec P 131e5ac1bcce424cbb1f7f60be20a263 reported cstate change: config changed from index 1270 to 1272, VOTER 5d1ee23344854913811909afd87ab724 (127.2.49.131) evicted. New cstate: current_term: 2 leader_uuid: "131e5ac1bcce424cbb1f7f60be20a263" committed_config { opid_index: 1272 OBSOLETE_local: false peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } } }
I20240503 08:13:55.718499  7834 tablet_service.cc:1514] Processing DeleteTablet for tablet 303fded976c342efbccfc375950f78ec with delete_type TABLET_DATA_TOMBSTONED (TS 5d1ee23344854913811909afd87ab724 not found in new config with opid_index 1272) from {username='slave'} at 127.0.0.1:37614
I20240503 08:13:55.719152 14135 tablet_replica.cc:330] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: stopping tablet replica
I20240503 08:13:55.719689 14135 raft_consensus.cc:2229] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:55.719892 14135 raft_consensus.cc:2258] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:55.721558 14135 ts_tablet_manager.cc:1875] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:55.727319 14135 ts_tablet_manager.cc:1888] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1271
I20240503 08:13:55.727480 14135 log.cc:1196] T 303fded976c342efbccfc375950f78ec P 5d1ee23344854913811909afd87ab724: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/303fded976c342efbccfc375950f78ec
I20240503 08:13:55.728415  7437 catalog_manager.cc:4830] TS 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): tablet 303fded976c342efbccfc375950f78ec (table rebalance_test_table_1 [id=82c2bc22dd4d462c9ad0635016e832a1]) successfully deleted
I20240503 08:13:55.856825  9297 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.866228  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:55.875720  7834 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:55.883725 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:55.884653  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:55.887382  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:55.900322  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:55.902913  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.117081  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.129452  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:56.138535  9297 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.141286  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:56.149541  7834 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:56.183012  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.226006 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:56.248943  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:56.603173  9039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.615242  7571 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:56.615331  9297 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.643311  7834 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:56.643445  7702 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:56.665875  9169 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.680951 13102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:56.689049  8900 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 56ef465a043845eb8a012eb5e7ce8be4 | 127.2.49.190:41099 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                               | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.2.49.148:33259 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 131e5ac1bcce424cbb1f7f60be20a263 | 127.2.49.135:37243 | HEALTHY | <none>   |       1        |       0
 1456c568b524465bb505956695d13738 | 127.2.49.133:39615 | HEALTHY | <none>   |       1        |       0
 359f2f323e79431eac07dc0a3f47a47b | 127.2.49.130:43861 | HEALTHY | <none>   |       2        |       0
 55fc889503854003801521bf97120179 | 127.2.49.129:39639 | HEALTHY | <none>   |       4        |       0
 5d1ee23344854913811909afd87ab724 | 127.2.49.131:40273 | HEALTHY | <none>   |       3        |       0
 b50f0dfea2284eacbe4184b1f3ae3753 | 127.2.49.134:37149 | HEALTHY | <none>   |       1        |       0
 fb1787264a5d40a1a72afabda025bc5e | 127.2.49.136:43675 | HEALTHY | <none>   |       0        |       0
 fb22180119ec428393402ddbd0b748b6 | 127.2.49.132:41615 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.49.129                                                                                                                                                    | experimental          | 127.2.49.129:39639
 local_ip_for_outbound_sockets            | 127.2.49.130                                                                                                                                                    | experimental          | 127.2.49.130:43861
 local_ip_for_outbound_sockets            | 127.2.49.131                                                                                                                                                    | experimental          | 127.2.49.131:40273
 local_ip_for_outbound_sockets            | 127.2.49.132                                                                                                                                                    | experimental          | 127.2.49.132:41615
 local_ip_for_outbound_sockets            | 127.2.49.133                                                                                                                                                    | experimental          | 127.2.49.133:39615
 local_ip_for_outbound_sockets            | 127.2.49.134                                                                                                                                                    | experimental          | 127.2.49.134:37149
 local_ip_for_outbound_sockets            | 127.2.49.135                                                                                                                                                    | experimental          | 127.2.49.135:37243
 local_ip_for_outbound_sockets            | 127.2.49.136                                                                                                                                                    | experimental          | 127.2.49.136:43675
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                           | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.49.129:39639
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.49.130:43861
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.49.131:40273
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.49.132:41615
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.2.49.133:39615
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.2.49.134:37149
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.2.49.135:37243
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.2.49.136:43675

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.2.49.148:33259 | all 8 server(s) checked
 time_source         | builtin            | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 08:13:56.770887  2246 log_verifier.cc:126] Checking tablet 01f87cf2ca7041cd8727e41d994f330c
I20240503 08:13:57.235448  2246 log_verifier.cc:177] Verified matching terms for 1378 ops in tablet 01f87cf2ca7041cd8727e41d994f330c
I20240503 08:13:57.237095  2246 log_verifier.cc:126] Checking tablet 0246c726f68848a7ad4620e533f93b53
I20240503 08:13:57.292610  2246 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 0246c726f68848a7ad4620e533f93b53
I20240503 08:13:57.292707  2246 log_verifier.cc:126] Checking tablet 0563aa33bb13450c96f620998b881c61
I20240503 08:13:57.648522  2246 log_verifier.cc:177] Verified matching terms for 1123 ops in tablet 0563aa33bb13450c96f620998b881c61
I20240503 08:13:57.649791  2246 log_verifier.cc:126] Checking tablet 303fded976c342efbccfc375950f78ec
I20240503 08:13:58.069433  2246 log_verifier.cc:177] Verified matching terms for 1272 ops in tablet 303fded976c342efbccfc375950f78ec
I20240503 08:13:58.107308  2246 log_verifier.cc:126] Checking tablet 3242894c953a4f6f8095bfb8c9318855
I20240503 08:13:58.509101  2246 log_verifier.cc:177] Verified matching terms for 1340 ops in tablet 3242894c953a4f6f8095bfb8c9318855
I20240503 08:13:58.510598  2246 log_verifier.cc:126] Checking tablet 44c94e75cff640f8927043b5e071c980
I20240503 08:13:58.945139  2246 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 44c94e75cff640f8927043b5e071c980
I20240503 08:13:58.946446  2246 log_verifier.cc:126] Checking tablet 6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:59.311951  2246 log_verifier.cc:177] Verified matching terms for 1181 ops in tablet 6462c70bafc94b7fb3bcb9502e80e096
I20240503 08:13:59.313472  2246 log_verifier.cc:126] Checking tablet 65167136cc2d478cb0efd6a374839c48
I20240503 08:13:59.699893  2246 log_verifier.cc:177] Verified matching terms for 1171 ops in tablet 65167136cc2d478cb0efd6a374839c48
I20240503 08:13:59.701464  2246 log_verifier.cc:126] Checking tablet 6c6b0211f6b44c1b92a5be884608d000
I20240503 08:14:00.050747  2246 log_verifier.cc:177] Verified matching terms for 1185 ops in tablet 6c6b0211f6b44c1b92a5be884608d000
I20240503 08:14:00.052107  2246 log_verifier.cc:126] Checking tablet 7e5a34b052e94b7b8a1dbaec2038d921
I20240503 08:14:00.514537  2246 log_verifier.cc:177] Verified matching terms for 1422 ops in tablet 7e5a34b052e94b7b8a1dbaec2038d921
I20240503 08:14:00.516124  2246 log_verifier.cc:126] Checking tablet 80eccc47e060493ebe43b007c7a23f00
I20240503 08:14:00.550107  7441 maintenance_manager.cc:419] P 56ef465a043845eb8a012eb5e7ce8be4: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033381
I20240503 08:14:00.577183  7440 maintenance_manager.cc:643] P 56ef465a043845eb8a012eb5e7ce8be4: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.026s	user 0.023s	sys 0.001s Metrics: {"bytes_written":12581,"cfile_init":1,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1132,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":667,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":25,"thread_start_us":518,"threads_started":2,"wal-append.queue_time_us":337}
I20240503 08:14:00.578420  7441 maintenance_manager.cc:419] P 56ef465a043845eb8a012eb5e7ce8be4: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3325 bytes on disk
I20240503 08:14:00.579169  7440 maintenance_manager.cc:643] P 56ef465a043845eb8a012eb5e7ce8be4: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:14:00.871021  2246 log_verifier.cc:177] Verified matching terms for 1191 ops in tablet 80eccc47e060493ebe43b007c7a23f00
I20240503 08:14:00.872308  2246 log_verifier.cc:126] Checking tablet 8967d6dac1aa482683c8bad1892dcbde
I20240503 08:14:01.304054  2246 log_verifier.cc:177] Verified matching terms for 1320 ops in tablet 8967d6dac1aa482683c8bad1892dcbde
I20240503 08:14:01.305552  2246 log_verifier.cc:126] Checking tablet 91ae2ad1ed4448689d8fe8eba7942dd9
I20240503 08:14:01.685983  2246 log_verifier.cc:177] Verified matching terms for 1262 ops in tablet 91ae2ad1ed4448689d8fe8eba7942dd9
I20240503 08:14:01.687425  2246 log_verifier.cc:126] Checking tablet 9f5a650ed26a406baa15105ac9fc8f8f
I20240503 08:14:02.084640  2246 log_verifier.cc:177] Verified matching terms for 1204 ops in tablet 9f5a650ed26a406baa15105ac9fc8f8f
I20240503 08:14:02.086374  2246 log_verifier.cc:126] Checking tablet a629b67231f84cdbb5e302265896e25a
I20240503 08:14:02.442404  2246 log_verifier.cc:177] Verified matching terms for 1165 ops in tablet a629b67231f84cdbb5e302265896e25a
I20240503 08:14:02.443926  2246 log_verifier.cc:126] Checking tablet a7f1cb831a9149e494cb31fac27f922d
I20240503 08:14:02.840476  2246 log_verifier.cc:177] Verified matching terms for 1209 ops in tablet a7f1cb831a9149e494cb31fac27f922d
I20240503 08:14:02.845227  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 7511
I20240503 08:14:02.866212  7632 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:03.166419  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 7511
W20240503 08:14:03.212735  7658 connection.cc:535] server connection from 127.2.49.129:39989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:03.212810 13058 connection.cc:535] server connection from 127.2.49.129:41573 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:03.213018  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 7642
I20240503 08:14:03.218541  7763 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:03.518886  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 7642
W20240503 08:14:03.569607  8854 connection.cc:535] client connection to 127.2.49.130:43861 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:03.569927  8854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:03.569963 13056 connection.cc:535] server connection from 127.2.49.130:40483 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:03.570425  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 7772
W20240503 08:14:03.572420  8854 consensus_peers.cc:485] T a7f1cb831a9149e494cb31fac27f922d P fb22180119ec428393402ddbd0b748b6 -> Peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): Couldn't send request to peer 359f2f323e79431eac07dc0a3f47a47b. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:43861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.576390  7893 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:03.892954  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 7772
W20240503 08:14:03.938867  8993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.49.131:40273 (error 108)
I20240503 08:14:03.939128  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 8834
W20240503 08:14:03.940968  8993 consensus_peers.cc:485] T 9f5a650ed26a406baa15105ac9fc8f8f P 1456c568b524465bb505956695d13738 -> Peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): Couldn't send request to peer 5d1ee23344854913811909afd87ab724. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:40273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:03.945212  8959 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:04.220152  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 8834
W20240503 08:14:04.268724  8993 connection.cc:535] server connection from 127.2.49.132:60165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:04.268714 13055 connection.cc:535] server connection from 127.2.49.132:47701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:04.269291  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 8970
I20240503 08:14:04.275141  9099 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:04.350275 14275 raft_consensus.cc:482] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 55fc889503854003801521bf97120179)
I20240503 08:14:04.350515 14275 raft_consensus.cc:504] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 728 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:14:04.351378 14275 leader_election.cc:288] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
W20240503 08:14:04.352497  9124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:39639: connect: Connection refused (error 111)
I20240503 08:14:04.355458 14291 raft_consensus.cc:482] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc889503854003801521bf97120179)
W20240503 08:14:04.355666  9124 leader_election.cc:334] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 55fc889503854003801521bf97120179 (127.2.49.129:39639): Network error: Client connection negotiation failed: client connection to 127.2.49.129:39639: connect: Connection refused (error 111)
I20240503 08:14:04.355636 14291 raft_consensus.cc:504] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:14:04.355791  9124 leader_election.cc:302] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b50f0dfea2284eacbe4184b1f3ae3753; no voters: 55fc889503854003801521bf97120179
I20240503 08:14:04.356063 14275 raft_consensus.cc:2726] T 8967d6dac1aa482683c8bad1892dcbde P b50f0dfea2284eacbe4184b1f3ae3753 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:14:04.356382 14291 leader_election.cc:288] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
W20240503 08:14:04.357292  9254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:39639: connect: Connection refused (error 111)
W20240503 08:14:04.358531  9254 leader_election.cc:334] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc889503854003801521bf97120179 (127.2.49.129:39639): Network error: Client connection negotiation failed: client connection to 127.2.49.129:39639: connect: Connection refused (error 111)
I20240503 08:14:04.358651  9254 leader_election.cc:302] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 131e5ac1bcce424cbb1f7f60be20a263; no voters: 55fc889503854003801521bf97120179
I20240503 08:14:04.358909 14291 raft_consensus.cc:2726] T 6c6b0211f6b44c1b92a5be884608d000 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:04.534509  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 8970
I20240503 08:14:04.580483  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 9107
I20240503 08:14:04.586784  9229 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:04.589382 14291 raft_consensus.cc:482] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d1ee23344854913811909afd87ab724)
I20240503 08:14:04.589579 14291 raft_consensus.cc:504] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:14:04.590042 14291 leader_election.cc:288] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
W20240503 08:14:04.591576  9253 leader_election.cc:334] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273): Network error: Client connection negotiation failed: client connection to 127.2.49.131:40273: connect: Connection refused (error 111)
I20240503 08:14:04.591713  9253 leader_election.cc:302] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 131e5ac1bcce424cbb1f7f60be20a263; no voters: 5d1ee23344854913811909afd87ab724
I20240503 08:14:04.591995 14291 raft_consensus.cc:2726] T a629b67231f84cdbb5e302265896e25a P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:04.668488 14303 raft_consensus.cc:482] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55fc889503854003801521bf97120179)
I20240503 08:14:04.668735 14303 raft_consensus.cc:504] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 1171 OBSOLETE_local: false peers { permanent_uuid: "55fc889503854003801521bf97120179" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 39639 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:14:04.670105 14303 leader_election.cc:288] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55fc889503854003801521bf97120179 (127.2.49.129:39639)
W20240503 08:14:04.671224 13057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:39639: connect: Connection refused (error 111)
W20240503 08:14:04.672659 13057 leader_election.cc:334] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55fc889503854003801521bf97120179 (127.2.49.129:39639): Network error: Client connection negotiation failed: client connection to 127.2.49.129:39639: connect: Connection refused (error 111)
I20240503 08:14:04.672839 13057 leader_election.cc:302] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fb1787264a5d40a1a72afabda025bc5e; no voters: 55fc889503854003801521bf97120179
I20240503 08:14:04.673182 14303 raft_consensus.cc:2726] T 65167136cc2d478cb0efd6a374839c48 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:04.854718 14275 raft_consensus.cc:482] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5d1ee23344854913811909afd87ab724)
I20240503 08:14:04.854933 14275 raft_consensus.cc:504] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "5d1ee23344854913811909afd87ab724" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 40273 } } peers { permanent_uuid: "b50f0dfea2284eacbe4184b1f3ae3753" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 37149 } }
I20240503 08:14:04.855608 14275 leader_election.cc:288] T 7e5a34b052e94b7b8a1dbaec2038d921 P b50f0dfea2284eacbe4184b1f3ae3753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d1ee23344854913811909afd87ab724 (127.2.49.131:40273)
I20240503 08:14:04.855825  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 9107
I20240503 08:14:04.903918  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 9237
I20240503 08:14:04.909826  9359 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:05.019421 14303 raft_consensus.cc:482] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 359f2f323e79431eac07dc0a3f47a47b)
I20240503 08:14:05.019613 14303 raft_consensus.cc:504] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 1340 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "fb1787264a5d40a1a72afabda025bc5e" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 43675 } }
I20240503 08:14:05.020378 14303 leader_election.cc:288] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861)
W20240503 08:14:05.021912 13058 leader_election.cc:334] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 359f2f323e79431eac07dc0a3f47a47b (127.2.49.130:43861): Network error: Client connection negotiation failed: client connection to 127.2.49.130:43861: connect: Connection refused (error 111)
I20240503 08:14:05.022089 13058 leader_election.cc:302] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fb1787264a5d40a1a72afabda025bc5e; no voters: 359f2f323e79431eac07dc0a3f47a47b
I20240503 08:14:05.022433 14303 raft_consensus.cc:2726] T 3242894c953a4f6f8095bfb8c9318855 P fb1787264a5d40a1a72afabda025bc5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:05.169998 14291 raft_consensus.cc:482] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 359f2f323e79431eac07dc0a3f47a47b)
I20240503 08:14:05.170421 14291 raft_consensus.cc:504] T 91ae2ad1ed4448689d8fe8eba7942dd9 P 131e5ac1bcce424cbb1f7f60be20a263 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "359f2f323e79431eac07dc0a3f47a47b" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 43861 } } peers { permanent_uuid: "131e5ac1bcce424cbb1f7f60be20a263" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 37243 } }
I20240503 08:14:05.171221  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 9237
I20240503 08:14:05.215179  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 13038
I20240503 08:14:05.221477 13161 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:05.417001  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 13038
I20240503 08:14:05.448654  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 7421
I20240503 08:14:05.449615  7482 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:05.694411  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 7421
2024-05-03T08:14:05Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/2 (125963 ms)
[----------] 1 test from RebalanceParamTest (125963 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240503 08:14:05.832618  2246 test_util.cc:268] Using random seed: 1302577549
I20240503 08:14:05.834604  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:14:05.834684  2246 ts_itest-base.cc:113] --------------
I20240503 08:14:05.834733  2246 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:14:05.834790  2246 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:14:05.834846  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:05Z Disabled control of system clock
I20240503 08:14:05.893368  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:42751
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:42751
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 08:14:06.251905 14318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:06.252249 14318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:06.252336 14318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:06.259891 14318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:06.260033 14318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:06.260083 14318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:06.260139 14318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:06.260180 14318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:06.269512 14318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:42751
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:06.270929 14318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:06.272693 14318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:06.283785 14323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:06.284346 14324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:06.285063 14326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:06.285984 14318 server_base.cc:1034] running on GCE node
I20240503 08:14:06.286881 14318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:06.288417 14318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:06.289626 14318 hybrid_clock.cc:648] HybridClock initialized: now 1714724046289583 us; error 43 us; skew 500 ppm
I20240503 08:14:06.289974 14318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:06.292419 14318 webserver.cc:460] Webserver started at http://127.2.49.190:44459/ using document root <none> and password file <none>
I20240503 08:14:06.293210 14318 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:06.293332 14318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:06.293660 14318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:06.296475 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "309eed6422ad4e71bcf73f518dc42423"
format_stamp: "Formatted at 2024-05-03 08:14:06 on dist-test-slave-dz87"
I20240503 08:14:06.297292 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "309eed6422ad4e71bcf73f518dc42423"
format_stamp: "Formatted at 2024-05-03 08:14:06 on dist-test-slave-dz87"
I20240503 08:14:06.302800 14318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 08:14:06.306321 14332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:06.307744 14318 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:14:06.307912 14318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "309eed6422ad4e71bcf73f518dc42423"
format_stamp: "Formatted at 2024-05-03 08:14:06 on dist-test-slave-dz87"
I20240503 08:14:06.308100 14318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:06.323338 14318 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:06.324193 14318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:06.324436 14318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:06.342989 14383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:42751 every 8 connection(s)
I20240503 08:14:06.342983 14318 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:42751
I20240503 08:14:06.344486 14318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:14:06.347506  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 14318
I20240503 08:14:06.347745  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:14:06.348775 14384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.361230 14384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423: Bootstrap starting.
I20240503 08:14:06.364136 14384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.365113 14384 log.cc:824] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:06.367799 14384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423: No bootstrap required, opened a new log
I20240503 08:14:06.373996 14384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309eed6422ad4e71bcf73f518dc42423" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42751 } }
I20240503 08:14:06.374470 14384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.374573 14384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309eed6422ad4e71bcf73f518dc42423, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.375183 14384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309eed6422ad4e71bcf73f518dc42423" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42751 } }
I20240503 08:14:06.375388 14384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:06.375510 14384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:06.375671 14384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.377779 14384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309eed6422ad4e71bcf73f518dc42423" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42751 } }
I20240503 08:14:06.378365 14384 leader_election.cc:302] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 309eed6422ad4e71bcf73f518dc42423; no voters: 
I20240503 08:14:06.379035 14384 leader_election.cc:288] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:14:06.379149 14389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:06.379817 14389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [term 1 LEADER]: Becoming Leader. State: Replica: 309eed6422ad4e71bcf73f518dc42423, State: Running, Role: LEADER
I20240503 08:14:06.380412 14389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309eed6422ad4e71bcf73f518dc42423" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42751 } }
I20240503 08:14:06.381136 14384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:06.383123 14390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "309eed6422ad4e71bcf73f518dc42423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309eed6422ad4e71bcf73f518dc42423" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42751 } } }
I20240503 08:14:06.383456 14390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:06.385700 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 309eed6422ad4e71bcf73f518dc42423. Latest consensus state: current_term: 1 leader_uuid: "309eed6422ad4e71bcf73f518dc42423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309eed6422ad4e71bcf73f518dc42423" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 42751 } } }
I20240503 08:14:06.386068 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:06.386948 14395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:06.392357 14395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:06.401208 14395 catalog_manager.cc:1347] Generated new cluster ID: 7e05a76fdd694057afa3c4a14984ce5c
I20240503 08:14:06.401350 14395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:06.419368 14395 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:06.420848 14395 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:06.439949 14395 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 309eed6422ad4e71bcf73f518dc42423: Generated new TSK 0
I20240503 08:14:06.441275 14395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:14:06.449658  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:14:06.800835 14408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:06.801144 14408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:06.801231 14408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:06.809491 14408 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:06.809725 14408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:06.809904 14408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:14:06.819540 14408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:06.820919 14408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:06.822731 14408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:06.835928 14413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:06.835947 14414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:06.836452 14416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:06.837108 14408 server_base.cc:1034] running on GCE node
I20240503 08:14:06.837728 14408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:06.838718 14408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:06.839910 14408 hybrid_clock.cc:648] HybridClock initialized: now 1714724046839862 us; error 39 us; skew 500 ppm
I20240503 08:14:06.840222 14408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:06.842485 14408 webserver.cc:460] Webserver started at http://127.2.49.129:42525/ using document root <none> and password file <none>
I20240503 08:14:06.843202 14408 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:06.843299 14408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:06.843581 14408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:06.846144 14408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
format_stamp: "Formatted at 2024-05-03 08:14:06 on dist-test-slave-dz87"
I20240503 08:14:06.846969 14408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
format_stamp: "Formatted at 2024-05-03 08:14:06 on dist-test-slave-dz87"
I20240503 08:14:06.852195 14408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:14:06.855651 14422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:06.857053 14408 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:14:06.857218 14408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
format_stamp: "Formatted at 2024-05-03 08:14:06 on dist-test-slave-dz87"
I20240503 08:14:06.857405 14408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:06.890319 14408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:06.891188 14408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:06.891435 14408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:06.892339 14408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:06.893886 14408 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:06.893983 14408 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:06.894057 14408 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:06.894110 14408 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:06.926998 14408 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:37829
I20240503 08:14:06.927117 14534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:37829 every 8 connection(s)
I20240503 08:14:06.928331 14408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:14:06.934283  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 14408
I20240503 08:14:06.934489  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:14:06.937608  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:06.941305 14535 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:06.941607 14535 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:06.942405 14535 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:06.944463 14349 ts_manager.cc:194] Registered new tserver with Master: 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:06.945909 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:59867
W20240503 08:14:07.293519 14539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:07.293890 14539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:07.294057 14539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:07.301954 14539 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:07.302105 14539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:07.302318 14539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:14:07.312287 14539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:07.313758 14539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:07.315704 14539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:07.327950 14544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.328256 14547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.328781 14545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:07.329505 14539 server_base.cc:1034] running on GCE node
I20240503 08:14:07.330206 14539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:07.331190 14539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:07.332381 14539 hybrid_clock.cc:648] HybridClock initialized: now 1714724047332334 us; error 41 us; skew 500 ppm
I20240503 08:14:07.332711 14539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:07.335103 14539 webserver.cc:460] Webserver started at http://127.2.49.130:34655/ using document root <none> and password file <none>
I20240503 08:14:07.335850 14539 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:07.335956 14539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:07.336283 14539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:07.338917 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0d77c4b1f12049dea71b97e86be2b288"
format_stamp: "Formatted at 2024-05-03 08:14:07 on dist-test-slave-dz87"
I20240503 08:14:07.339689 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0d77c4b1f12049dea71b97e86be2b288"
format_stamp: "Formatted at 2024-05-03 08:14:07 on dist-test-slave-dz87"
I20240503 08:14:07.345571 14539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240503 08:14:07.349933 14553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:07.351574 14539 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:14:07.351780 14539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0d77c4b1f12049dea71b97e86be2b288"
format_stamp: "Formatted at 2024-05-03 08:14:07 on dist-test-slave-dz87"
I20240503 08:14:07.352098 14539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:07.388602 14539 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:07.389497 14539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:07.389770 14539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:07.390836 14539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:07.392414 14539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:07.392508 14539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:07.392630 14539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:07.392704 14539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:07.424194 14539 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:35021
I20240503 08:14:07.424249 14665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:35021 every 8 connection(s)
I20240503 08:14:07.425546 14539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:14:07.431952  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 14539
I20240503 08:14:07.432154  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:14:07.435345  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:07.438956 14666 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:07.439301 14666 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:07.440212 14666 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:07.441720 14349 ts_manager.cc:194] Registered new tserver with Master: 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:07.442534 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:44725
W20240503 08:14:07.783250 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:07.783551 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:07.783669 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:07.791185 14669 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:07.791313 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:07.791491 14669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:14:07.800592 14669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:07.801944 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:07.803704 14669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:07.816200 14674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.816731 14677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.817332 14675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:07.817979 14669 server_base.cc:1034] running on GCE node
I20240503 08:14:07.818598 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:07.819505 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:07.820703 14669 hybrid_clock.cc:648] HybridClock initialized: now 1714724047820654 us; error 45 us; skew 500 ppm
I20240503 08:14:07.821003 14669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:07.823223 14669 webserver.cc:460] Webserver started at http://127.2.49.131:32959/ using document root <none> and password file <none>
I20240503 08:14:07.823915 14669 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:07.824030 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:07.824362 14669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:07.826941 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
format_stamp: "Formatted at 2024-05-03 08:14:07 on dist-test-slave-dz87"
I20240503 08:14:07.827704 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
format_stamp: "Formatted at 2024-05-03 08:14:07 on dist-test-slave-dz87"
I20240503 08:14:07.833078 14669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 08:14:07.836613 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:07.838004 14669 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:07.838217 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
format_stamp: "Formatted at 2024-05-03 08:14:07 on dist-test-slave-dz87"
I20240503 08:14:07.838420 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:07.857079 14669 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:07.857981 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:07.858358 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:07.859462 14669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:07.861008 14669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:07.861095 14669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:07.861187 14669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:07.861241 14669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:07.893843 14669 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:45663
I20240503 08:14:07.893898 14795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:45663 every 8 connection(s)
I20240503 08:14:07.895193 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:14:07.901681  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 14669
I20240503 08:14:07.901890  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:14:07.905468  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:07.913215 14796 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:07.913569 14796 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:07.914383 14796 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:07.915854 14349 ts_manager.cc:194] Registered new tserver with Master: c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:07.917078 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:44009
I20240503 08:14:07.948886 14535 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
W20240503 08:14:08.253314 14799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:08.253597 14799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:08.253717 14799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:08.261679 14799 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:08.261840 14799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:08.262073 14799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:14:08.271597 14799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:08.272977 14799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:08.274796 14799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:08.287775 14804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:08.288304 14805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:08.289001 14807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:08.289759 14799 server_base.cc:1034] running on GCE node
I20240503 08:14:08.290463 14799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:08.291409 14799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:08.292601 14799 hybrid_clock.cc:648] HybridClock initialized: now 1714724048292547 us; error 47 us; skew 500 ppm
I20240503 08:14:08.292903 14799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:08.295215 14799 webserver.cc:460] Webserver started at http://127.2.49.132:37037/ using document root <none> and password file <none>
I20240503 08:14:08.295926 14799 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:08.296036 14799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:08.296360 14799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:08.299052 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:08.299850 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:08.305136 14799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 08:14:08.308595 14813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:08.310292 14799 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:14:08.310473 14799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:08.310679 14799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:08.327337 14799 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:08.328210 14799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:08.328485 14799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:08.329449 14799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:08.331146 14799 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:08.331245 14799 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:08.331357 14799 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:08.331431 14799 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:08.367964 14799 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:35075
I20240503 08:14:08.368007 14925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:35075 every 8 connection(s)
I20240503 08:14:08.369302 14799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:14:08.371659  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 14799
I20240503 08:14:08.371847  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:14:08.375032  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:0
--local_ip_for_outbound_sockets=127.2.49.133
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:08.382294 14926 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:08.382601 14926 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:08.383373 14926 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:08.384843 14349 ts_manager.cc:194] Registered new tserver with Master: b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:08.385674 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:60777
I20240503 08:14:08.445567 14666 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
W20240503 08:14:08.726449 14929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:08.726740 14929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:08.726828 14929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:08.734499 14929 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:08.734632 14929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:08.734756 14929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:14:08.744108 14929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:08.745383 14929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:08.747079 14929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:08.759328 14934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:08.760066 14935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:08.760494 14937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:08.761554 14929 server_base.cc:1034] running on GCE node
I20240503 08:14:08.762199 14929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:08.763252 14929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:08.764448 14929 hybrid_clock.cc:648] HybridClock initialized: now 1714724048764395 us; error 53 us; skew 500 ppm
I20240503 08:14:08.764760 14929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:08.767141 14929 webserver.cc:460] Webserver started at http://127.2.49.133:36529/ using document root <none> and password file <none>
I20240503 08:14:08.767894 14929 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:08.768013 14929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:08.768326 14929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:08.771008 14929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:08.771804 14929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:08.777037 14929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20240503 08:14:08.780547 14943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:08.782205 14929 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:14:08.782387 14929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:08.782585 14929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:08.808679 14929 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:08.809597 14929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:08.809859 14929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:08.810863 14929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:08.812350 14929 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:08.812435 14929 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:08.812538 14929 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:08.812618 14929 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:08.845373 14929 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:38023
I20240503 08:14:08.845413 15055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:38023 every 8 connection(s)
I20240503 08:14:08.846833 14929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:14:08.849630  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 14929
I20240503 08:14:08.849843  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:14:08.853365  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.134:0
--local_ip_for_outbound_sockets=127.2.49.134
--webserver_interface=127.2.49.134
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:08.861276 15056 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:08.861596 15056 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:08.862455 15056 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:08.863974 14349 ts_manager.cc:194] Registered new tserver with Master: 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:14:08.865099 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:37301
I20240503 08:14:08.920178 14796 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
W20240503 08:14:09.213986 15059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:09.214408 15059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:09.214586 15059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:09.222551 15059 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:09.222723 15059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:09.222893 15059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.134
I20240503 08:14:09.232405 15059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.49.134
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.134
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:09.233847 15059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:09.235718 15059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:09.248651 15067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:09.249117 15064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:09.249639 15065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:09.250568 15059 server_base.cc:1034] running on GCE node
I20240503 08:14:09.251195 15059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:09.252229 15059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:09.253450 15059 hybrid_clock.cc:648] HybridClock initialized: now 1714724049253400 us; error 42 us; skew 500 ppm
I20240503 08:14:09.253775 15059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:09.256132 15059 webserver.cc:460] Webserver started at http://127.2.49.134:35299/ using document root <none> and password file <none>
I20240503 08:14:09.256871 15059 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:09.256990 15059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:09.257313 15059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:09.260035 15059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "95860e22c9474e2d894f04deba8442cc"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:09.260820 15059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "95860e22c9474e2d894f04deba8442cc"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:09.266206 15059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240503 08:14:09.271499 15073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:09.272827 15059 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20240503 08:14:09.272997 15059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "95860e22c9474e2d894f04deba8442cc"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:09.273206 15059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:09.290624 15059 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:09.291496 15059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:09.291762 15059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:09.292717 15059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:09.294276 15059 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:09.294374 15059 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:09.294487 15059 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:09.294562 15059 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:09.327400 15059 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.134:36937
I20240503 08:14:09.327469 15185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.134:36937 every 8 connection(s)
I20240503 08:14:09.328703 15059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:14:09.331348  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 15059
I20240503 08:14:09.331558  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:14:09.335127  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.135:0
--local_ip_for_outbound_sockets=127.2.49.135
--webserver_interface=127.2.49.135
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:09.342664 15186 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:09.343091 15186 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:09.344044 15186 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:09.345718 14349 ts_manager.cc:194] Registered new tserver with Master: 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937)
I20240503 08:14:09.346889 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.134:48689
I20240503 08:14:09.388427 14926 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
W20240503 08:14:09.687635 15189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:09.687958 15189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:09.688063 15189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:09.695588 15189 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:09.695724 15189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:09.695848 15189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.135
I20240503 08:14:09.704857 15189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.49.135
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.135
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:09.706179 15189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:09.707831 15189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:09.720275 15194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:09.720854 15195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:09.721446 15197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:09.722285 15189 server_base.cc:1034] running on GCE node
I20240503 08:14:09.722911 15189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:09.723903 15189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:09.725221 15189 hybrid_clock.cc:648] HybridClock initialized: now 1714724049725159 us; error 46 us; skew 500 ppm
I20240503 08:14:09.725548 15189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:09.727878 15189 webserver.cc:460] Webserver started at http://127.2.49.135:36317/ using document root <none> and password file <none>
I20240503 08:14:09.728603 15189 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:09.728721 15189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:09.729033 15189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:09.731659 15189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6835ad958e094ce4927135b942a2e8a8"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:09.732429 15189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6835ad958e094ce4927135b942a2e8a8"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:09.737602 15189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.006s
I20240503 08:14:09.741094 15203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:09.742458 15189 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:14:09.742609 15189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6835ad958e094ce4927135b942a2e8a8"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:09.742833 15189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:09.758527 15189 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:09.759372 15189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:09.759665 15189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:09.760620 15189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:09.762147 15189 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:09.762291 15189 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:09.762401 15189 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:09.762475 15189 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:09.793291 15189 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.135:46299
I20240503 08:14:09.793350 15315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.135:46299 every 8 connection(s)
I20240503 08:14:09.794606 15189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:14:09.802490  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 15189
I20240503 08:14:09.802675  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:14:09.805848  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.136:0
--local_ip_for_outbound_sockets=127.2.49.136
--webserver_interface=127.2.49.136
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:09.807504 15316 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:09.807855 15316 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:09.808815 15316 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:09.810319 14349 ts_manager.cc:194] Registered new tserver with Master: 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:14:09.811208 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.135:42851
I20240503 08:14:09.868283 15056 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
W20240503 08:14:10.160244 15319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:10.160547 15319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:10.160637 15319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:10.168233 15319 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:10.168366 15319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:10.168491 15319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.136
I20240503 08:14:10.177548 15319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.49.136
--webserver_port=0
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.136
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:10.178928 15319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:10.180812 15319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:10.193210 15325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:10.193658 15327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:10.194203 15324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:10.195145 15319 server_base.cc:1034] running on GCE node
I20240503 08:14:10.195735 15319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:10.196686 15319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:10.197896 15319 hybrid_clock.cc:648] HybridClock initialized: now 1714724050197843 us; error 40 us; skew 500 ppm
I20240503 08:14:10.198271 15319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:10.200474 15319 webserver.cc:460] Webserver started at http://127.2.49.136:40021/ using document root <none> and password file <none>
I20240503 08:14:10.201222 15319 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:10.201337 15319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:10.201653 15319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:10.204267 15319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "036223b0890d4634acfadf9596f39f80"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-dz87"
I20240503 08:14:10.205031 15319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "036223b0890d4634acfadf9596f39f80"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-dz87"
I20240503 08:14:10.210249 15319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.003s
I20240503 08:14:10.213671 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:10.215008 15319 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:14:10.215189 15319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "036223b0890d4634acfadf9596f39f80"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-dz87"
I20240503 08:14:10.215382 15319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:10.240942 15319 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:10.241791 15319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:10.242081 15319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:10.243117 15319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:10.244571 15319 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:10.244671 15319 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:10.244787 15319 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:10.244858 15319 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:10.276587 15319 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.136:37641
I20240503 08:14:10.276638 15445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.136:37641 every 8 connection(s)
I20240503 08:14:10.277967 15319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:14:10.282385  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 15319
I20240503 08:14:10.282599  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:14:10.291035 15446 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:10.291308 15446 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:10.292001 15446 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:10.293329 14349 ts_manager.cc:194] Registered new tserver with Master: 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:14:10.294124 14349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.136:57499
I20240503 08:14:10.298619  2246 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:14:10.315026  2246 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:14:10.315201  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14929
I20240503 08:14:10.323356 15051 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:10.350003 15186 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
I20240503 08:14:10.409461  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14929
I20240503 08:14:10.426537  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 15059
I20240503 08:14:10.434592 15181 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:10.516031  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 15059
I20240503 08:14:10.532577  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 15189
I20240503 08:14:10.540813 15311 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:10.622020  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 15189
I20240503 08:14:10.638785  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 15319
I20240503 08:14:10.647243 15441 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:10.727836  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 15319
I20240503 08:14:14.496253 14349 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:14.534595 14600 tablet_service.cc:1467] Processing CreateTablet for tablet f65ab6dba1124573962e2da58aa1859c (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.536142 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65ab6dba1124573962e2da58aa1859c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.538589 14601 tablet_service.cc:1467] Processing CreateTablet for tablet 6b677894b51f4dc58f157d8f4d632e9a (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.539080 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b677894b51f4dc58f157d8f4d632e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.539888 14470 tablet_service.cc:1467] Processing CreateTablet for tablet 6b677894b51f4dc58f157d8f4d632e9a (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.541538 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b677894b51f4dc58f157d8f4d632e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.542119 14469 tablet_service.cc:1467] Processing CreateTablet for tablet f65ab6dba1124573962e2da58aa1859c (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.542605 14468 tablet_service.cc:1467] Processing CreateTablet for tablet dc9575f64eb741948e51ccf06f48228a (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.544332 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65ab6dba1124573962e2da58aa1859c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.544597 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9575f64eb741948e51ccf06f48228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.545076 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 6b677894b51f4dc58f157d8f4d632e9a (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.545157 14860 tablet_service.cc:1467] Processing CreateTablet for tablet dc9575f64eb741948e51ccf06f48228a (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.547281 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b677894b51f4dc58f157d8f4d632e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.547784 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9575f64eb741948e51ccf06f48228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.560813 15475 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:14.565055 15475 tablet_bootstrap.cc:654] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.566514 15475 log.cc:824] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:14.566612 15476 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:14.570607 15476 tablet_bootstrap.cc:654] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.572090 15476 log.cc:824] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:14.573405 15478 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:14.575788 15475 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:14.576102 15475 ts_tablet_manager.cc:1367] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.009s
I20240503 08:14:14.577068 15478 tablet_bootstrap.cc:654] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.578054 15478 log.cc:824] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:14.577565 14731 tablet_service.cc:1467] Processing CreateTablet for tablet f65ab6dba1124573962e2da58aa1859c (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.579138 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65ab6dba1124573962e2da58aa1859c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.582923 14730 tablet_service.cc:1467] Processing CreateTablet for tablet dc9575f64eb741948e51ccf06f48228a (DEFAULT_TABLE table=rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:14.583530 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9575f64eb741948e51ccf06f48228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.583669 15476 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:14.584005 15476 ts_tablet_manager.cc:1367] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.000s
I20240503 08:14:14.584321 15475 raft_consensus.cc:348] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.584674 15475 raft_consensus.cc:374] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.584821 15475 raft_consensus.cc:729] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.585469 15475 consensus_queue.cc:260] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.594190 15475 ts_tablet_manager.cc:1398] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.018s	user 0.011s	sys 0.007s
I20240503 08:14:14.593955 15476 raft_consensus.cc:348] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.594751 15476 raft_consensus.cc:374] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.594929 15476 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.595917 15476 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.596290 15475 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:14.599001 15475 tablet_bootstrap.cc:654] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.602627 15478 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:14.603052 15478 ts_tablet_manager.cc:1367] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.030s	user 0.008s	sys 0.000s
I20240503 08:14:14.603865 15483 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:14.606840 15483 tablet_bootstrap.cc:654] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.607892 15483 log.cc:824] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:14.609393 15476 ts_tablet_manager.cc:1398] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.025s	user 0.017s	sys 0.000s
I20240503 08:14:14.609922 15476 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:14.611948 15476 tablet_bootstrap.cc:654] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.613579 15478 raft_consensus.cc:348] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.614378 15478 raft_consensus.cc:374] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.614542 15478 raft_consensus.cc:729] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.615583 15478 consensus_queue.cc:260] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.617158 15475 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:14.617391 15475 ts_tablet_manager.cc:1367] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.014s
I20240503 08:14:14.618628 15475 raft_consensus.cc:348] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.618858 15475 raft_consensus.cc:374] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.618950 15475 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.619290 15478 ts_tablet_manager.cc:1398] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.016s	user 0.014s	sys 0.000s
I20240503 08:14:14.619211 15475 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.619763 15478 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:14.619899 15475 ts_tablet_manager.cc:1398] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:14:14.621177 15476 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:14.621353 15476 ts_tablet_manager.cc:1367] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240503 08:14:14.621560 15478 tablet_bootstrap.cc:654] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.622498 15476 raft_consensus.cc:348] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.622807 15476 raft_consensus.cc:374] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.622933 15476 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.623255 15476 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.623984 15476 ts_tablet_manager.cc:1398] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:14.624385 15476 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
W20240503 08:14:14.627184 14927 tablet.cc:2369] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:14.626793 15476 tablet_bootstrap.cc:654] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.627643 15482 raft_consensus.cc:482] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:14.628610 15482 raft_consensus.cc:504] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.631297 15490 raft_consensus.cc:482] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:14.631481 15490 raft_consensus.cc:504] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.632529 15482 leader_election.cc:288] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:14.632822 15483 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:14.633181 15483 ts_tablet_manager.cc:1367] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.030s	user 0.014s	sys 0.000s
I20240503 08:14:14.633554 15490 leader_election.cc:288] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:14.641268 15476 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:14.641530 15476 ts_tablet_manager.cc:1367] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.000s
I20240503 08:14:14.642660 15476 raft_consensus.cc:348] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.643019 15476 raft_consensus.cc:374] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.643112 15476 raft_consensus.cc:729] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.643452 15476 consensus_queue.cc:260] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.644250 15476 ts_tablet_manager.cc:1398] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:14:14.646817 15478 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:14.647006 15478 ts_tablet_manager.cc:1367] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.027s	user 0.004s	sys 0.006s
I20240503 08:14:14.647753 15483 raft_consensus.cc:348] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.648061 15478 raft_consensus.cc:348] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.648270 15478 raft_consensus.cc:374] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.648293 15483 raft_consensus.cc:374] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.648370 15478 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.648486 15483 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.648700 15478 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.649351 15478 ts_tablet_manager.cc:1398] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240503 08:14:14.649458 15483 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.661830 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:14.663098 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:14.663548 14751 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:14.663507 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
W20240503 08:14:14.663645 14426 leader_election.cc:341] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:14.664013 14881 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:14.664386 14426 leader_election.cc:302] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:14.664633 14424 leader_election.cc:302] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:14:14.665083 15482 raft_consensus.cc:2781] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:14.665268 15482 raft_consensus.cc:482] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:14.665369 15490 raft_consensus.cc:2781] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:14.665402 15482 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.665505 15490 raft_consensus.cc:482] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:14.665607 15490 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.668793 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:14.668982 15490 raft_consensus.cc:504] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.669370 14621 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:14.669721 15490 leader_election.cc:288] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:14.670109 15482 raft_consensus.cc:504] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.670373 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:14.670563 14621 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.670984 15482 leader_election.cc:288] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:14.671578 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:14.671790 14881 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.673635 14621 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:14.674118 14423 leader_election.cc:302] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:14.674446 15482 raft_consensus.cc:2781] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:14.674660 15482 raft_consensus.cc:686] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:14.674809 14881 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:14.675258 14424 leader_election.cc:302] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:14.675259 15482 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:14.675890 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:14.676162 14751 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.675856 15490 raft_consensus.cc:2781] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:14.676582 15490 raft_consensus.cc:686] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:14.676879 15490 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.679824 14751 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:14.680445 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
W20240503 08:14:14.682718 14667 tablet.cc:2369] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:14:14.683799 14426 leader_election.cc:341] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:14.684626 15483 ts_tablet_manager.cc:1398] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.051s	user 0.021s	sys 0.025s
W20240503 08:14:14.685199 14536 tablet.cc:2369] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:14.686137 15483 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:14.687880 15483 tablet_bootstrap.cc:654] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.690265 15483 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:14.690413 15483 ts_tablet_manager.cc:1367] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.003s
I20240503 08:14:14.690033 14349 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:14:14.691329 15483 raft_consensus.cc:348] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.691491 15483 raft_consensus.cc:374] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.691576 15483 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.691793 15483 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.692265 15483 ts_tablet_manager.cc:1398] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:14.695027 14348 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:14.893077 15480 raft_consensus.cc:482] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:14.893277 15480 raft_consensus.cc:504] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.895031 15480 leader_election.cc:288] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:14.897955 15495 raft_consensus.cc:482] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:14.898133 15495 raft_consensus.cc:504] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.899098 15495 leader_election.cc:288] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:14.899556 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b677894b51f4dc58f157d8f4d632e9a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:14.899799 14621 raft_consensus.cc:2454] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:14.899804 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b677894b51f4dc58f157d8f4d632e9a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:14.900059 14881 raft_consensus.cc:2454] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:14.900264 14423 leader_election.cc:302] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:14.901319 15495 raft_consensus.cc:2781] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:14.901435 15495 raft_consensus.cc:482] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:14.901520 15495 raft_consensus.cc:3037] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:14:14.901945 14797 tablet.cc:2369] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:14.904423 15495 raft_consensus.cc:504] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.905622 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b677894b51f4dc58f157d8f4d632e9a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:14.905889 14621 raft_consensus.cc:3037] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.905903 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b677894b51f4dc58f157d8f4d632e9a" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:14.906399 14490 raft_consensus.cc:2379] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d77c4b1f12049dea71b97e86be2b288 in current term 1: Already voted for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in this term.
I20240503 08:14:14.907083 15495 leader_election.cc:288] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:14.907027 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b677894b51f4dc58f157d8f4d632e9a" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:14.907292 14881 raft_consensus.cc:2454] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:14.907835 14555 leader_election.cc:302] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:14.908195 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b677894b51f4dc58f157d8f4d632e9a" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:14.908390 14881 raft_consensus.cc:3037] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.908725 14621 raft_consensus.cc:2454] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:14.908869 15480 raft_consensus.cc:2741] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 08:14:14.909122 14423 leader_election.cc:302] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:14.909413 15495 raft_consensus.cc:2781] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:14.909668 15495 raft_consensus.cc:686] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:14.909969 15495 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:14.911058 14881 raft_consensus.cc:2454] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:14.912832 14349 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:15.112361 15494 raft_consensus.cc:482] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:15.112572 15494 raft_consensus.cc:504] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.114276 15494 leader_election.cc:288] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:15.122648 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:15.123025 14881 raft_consensus.cc:2379] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in current term 1: Already voted for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in this term.
I20240503 08:14:15.123317 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:15.123950 14687 leader_election.cc:302] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:14:15.124348 15494 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.126401 15494 raft_consensus.cc:2726] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:15.211598 15495 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:15.220382 15495 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:15.240048 15495 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:15.245817 15482 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:15.357733 15502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.358075 15502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.365774 15502 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:14:15.474634 15511 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:15.490847 15511 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 252.128 ms
  time per row: 0.126064 ms
I20240503 08:14:15.776811 14348 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:15.788949 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 55067e08fb694e90b6366d41cc962095 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.789476 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.789809 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 169bd9ea6772420abf757dd55cf44d31 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.790263 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.789074 14601 tablet_service.cc:1467] Processing CreateTablet for tablet 169bd9ea6772420abf757dd55cf44d31 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.794984 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.795450 14731 tablet_service.cc:1467] Processing CreateTablet for tablet 169bd9ea6772420abf757dd55cf44d31 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.795485 14729 tablet_service.cc:1467] Processing CreateTablet for tablet dd11460051ab40febc2fe6aff58b5028 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.795898 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.802750 14730 tablet_service.cc:1467] Processing CreateTablet for tablet 55067e08fb694e90b6366d41cc962095 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.803309 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.803380 14468 tablet_service.cc:1467] Processing CreateTablet for tablet dd11460051ab40febc2fe6aff58b5028 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.803517 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 55067e08fb694e90b6366d41cc962095 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.803783 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd11460051ab40febc2fe6aff58b5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.804083 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.805677 15570 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:15.807674 15570 tablet_bootstrap.cc:654] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.807633 14599 tablet_service.cc:1467] Processing CreateTablet for tablet dd11460051ab40febc2fe6aff58b5028 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:15.808121 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd11460051ab40febc2fe6aff58b5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.809667 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd11460051ab40febc2fe6aff58b5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.810511 15572 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:15.815070 15572 tablet_bootstrap.cc:654] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.817010 15573 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:15.818818 15569 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:15.820180 15573 tablet_bootstrap.cc:654] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.820819 15569 tablet_bootstrap.cc:654] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.822129 15572 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:15.823096 15572 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.004s
I20240503 08:14:15.824299 15572 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.825632 15570 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:15.825837 15570 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.003s
I20240503 08:14:15.826802 15570 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.827044 15570 raft_consensus.cc:374] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.827143 15570 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.827428 15570 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.827769 15569 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:15.827932 15569 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.008s
I20240503 08:14:15.827993 15570 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:14:15.828279 15570 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:15.828253 15573 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:15.828847 15573 ts_tablet_manager.cc:1367] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.004s
I20240503 08:14:15.828800 15569 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.829010 15569 raft_consensus.cc:374] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.829106 15569 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.829420 15569 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.829967 15573 raft_consensus.cc:348] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.830137 15569 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240503 08:14:15.830308 15573 raft_consensus.cc:374] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.830471 15569 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:15.830461 15573 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.830662 15572 raft_consensus.cc:374] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.830818 15572 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.830935 15573 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.831236 15570 tablet_bootstrap.cc:654] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.832175 15569 tablet_bootstrap.cc:654] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.833122 15572 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.833791 15573 ts_tablet_manager.cc:1398] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 08:14:15.838369 15572 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.000s
I20240503 08:14:15.838711 15572 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:15.838948 15569 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:15.839111 15569 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240503 08:14:15.839856 15569 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.840081 15569 raft_consensus.cc:374] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.840150 15569 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.840435 15572 tablet_bootstrap.cc:654] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.840416 15569 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.840934 15569 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:14:15.841233 15569 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:15.842856 15569 tablet_bootstrap.cc:654] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.845319 15570 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:15.845525 15570 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.001s
I20240503 08:14:15.846392 15570 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.846661 15570 raft_consensus.cc:374] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.846746 15570 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.847155 15572 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:15.847035 15570 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.847244 15569 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:15.847327 15572 ts_tablet_manager.cc:1367] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.001s
I20240503 08:14:15.847393 15569 ts_tablet_manager.cc:1367] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.002s
I20240503 08:14:15.847726 15570 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:14:15.848047 15569 raft_consensus.cc:348] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.848222 15569 raft_consensus.cc:374] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.848317 15569 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.848569 15569 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.849153 15569 ts_tablet_manager.cc:1398] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:14:15.849453 15572 raft_consensus.cc:348] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.850364 15572 raft_consensus.cc:374] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.850458 15572 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.850725 15572 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.851276 15572 ts_tablet_manager.cc:1398] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:14:15.851657 15572 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:15.853294 15572 tablet_bootstrap.cc:654] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.855156 15572 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:15.855289 15572 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:14:15.855777 15480 raft_consensus.cc:482] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:15.855857 15572 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.855921 15480 raft_consensus.cc:504] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.856035 15572 raft_consensus.cc:374] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.856132 15572 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.856340 15572 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.856587 15480 leader_election.cc:288] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:15.856801 15572 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:14:15.857456 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:15.857766 14490 raft_consensus.cc:2454] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:15.858511 14557 leader_election.cc:302] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:15.858850 15480 raft_consensus.cc:2781] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:15.858963 15480 raft_consensus.cc:482] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:15.859084 15480 raft_consensus.cc:3037] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.861199 15480 raft_consensus.cc:504] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.861943 15480 leader_election.cc:288] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:15.862341 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:15.862545 14490 raft_consensus.cc:3037] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.864233 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:15.864367 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:15.864439 14749 raft_consensus.cc:2454] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:15.864912 14490 raft_consensus.cc:2454] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:15.865235 14557 leader_election.cc:302] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:14:15.865526 15480 raft_consensus.cc:2781] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:15.865931 15480 raft_consensus.cc:686] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:15.866475 15480 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:15.869415 14342 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:14:15.871905 15480 raft_consensus.cc:482] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:15.872144 15480 raft_consensus.cc:504] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.872867 15480 leader_election.cc:288] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:15.873339 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:15.873409 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:15.873564 14751 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:15.873631 14879 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:15.873970 14557 leader_election.cc:302] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:15.874327 15480 raft_consensus.cc:2781] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:15.874425 15480 raft_consensus.cc:482] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:15.874536 15480 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.876643 15480 raft_consensus.cc:504] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.877379 15480 leader_election.cc:288] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:15.877802 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:15.878049 14751 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.878136 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:15.878381 14879 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.880180 14751 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:15.880389 14879 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:15.880854 14555 leader_election.cc:302] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:15.881275 15480 raft_consensus.cc:2781] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:15.881471 15480 raft_consensus.cc:686] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:15.881750 15480 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:15.884500 14343 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:15.969681 15512 raft_consensus.cc:482] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:15.969936 15512 raft_consensus.cc:504] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.971585 15512 leader_election.cc:288] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:15.980769 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55067e08fb694e90b6366d41cc962095" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:15.980813 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55067e08fb694e90b6366d41cc962095" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:15.981060 14620 raft_consensus.cc:2454] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:15.981081 14751 raft_consensus.cc:2454] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:15.981586 14817 leader_election.cc:302] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:15.982000 15512 raft_consensus.cc:2781] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:15.982129 15512 raft_consensus.cc:482] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:15.982250 15512 raft_consensus.cc:3037] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.984272 15512 raft_consensus.cc:504] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.984902 15512 leader_election.cc:288] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:15.985422 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55067e08fb694e90b6366d41cc962095" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:15.985436 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55067e08fb694e90b6366d41cc962095" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:15.985620 14751 raft_consensus.cc:3037] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.985635 14620 raft_consensus.cc:3037] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.987695 14620 raft_consensus.cc:2454] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:15.987705 14751 raft_consensus.cc:2454] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:15.988155 14814 leader_election.cc:302] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:15.988539 15512 raft_consensus.cc:2781] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:15.989017 15512 raft_consensus.cc:686] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:14:15.989496 15512 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:15.992697 14343 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:14:16.283686 14751 raft_consensus.cc:3037] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:16.286514 15480 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:16.297111 15480 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:16.335379 15597 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:16.336275 15597 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:16.384302 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.384680 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.393074 15593 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:14:16.437304 15512 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:16.446225 15591 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 234.357 ms
  time per row: 0.117179 ms
I20240503 08:14:16.787287 14343 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:16.795948 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 3fea213e5a12452f8b894b9cdb94f9ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.795941 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 3fea213e5a12452f8b894b9cdb94f9ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.796399 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea213e5a12452f8b894b9cdb94f9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.796398 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea213e5a12452f8b894b9cdb94f9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.797250 14469 tablet_service.cc:1467] Processing CreateTablet for tablet 12df78f752c34fe3885b235c4a19e325 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.797443 14730 tablet_service.cc:1467] Processing CreateTablet for tablet 12df78f752c34fe3885b235c4a19e325 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.797695 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12df78f752c34fe3885b235c4a19e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.797827 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12df78f752c34fe3885b235c4a19e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.798548 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 12df78f752c34fe3885b235c4a19e325 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.798908 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12df78f752c34fe3885b235c4a19e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.802588 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 3fea213e5a12452f8b894b9cdb94f9ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.803035 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea213e5a12452f8b894b9cdb94f9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.803056 14599 tablet_service.cc:1467] Processing CreateTablet for tablet 30b13cdf4b234f17872f5d1588915968 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.803458 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b13cdf4b234f17872f5d1588915968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.807258 14470 tablet_service.cc:1467] Processing CreateTablet for tablet 30b13cdf4b234f17872f5d1588915968 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.807770 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b13cdf4b234f17872f5d1588915968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.813102 15656 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:16.813585 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 30b13cdf4b234f17872f5d1588915968 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.814102 15657 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:16.814123 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b13cdf4b234f17872f5d1588915968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.815639 15656 tablet_bootstrap.cc:654] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.815889 15657 tablet_bootstrap.cc:654] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.821195 15660 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:16.825498 15660 tablet_bootstrap.cc:654] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.825923 15657 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:16.826067 15655 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:16.826094 15657 ts_tablet_manager.cc:1367] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240503 08:14:16.826720 15656 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:16.826941 15656 ts_tablet_manager.cc:1367] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.000s
I20240503 08:14:16.826978 15657 raft_consensus.cc:348] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.827203 15657 raft_consensus.cc:374] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.827316 15657 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.827579 15657 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.828099 15656 raft_consensus.cc:348] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.828348 15657 ts_tablet_manager.cc:1398] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:14:16.828361 15656 raft_consensus.cc:374] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.828569 15655 tablet_bootstrap.cc:654] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.828554 15656 raft_consensus.cc:729] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.828608 15657 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:16.828904 15656 consensus_queue.cc:260] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.829679 15656 ts_tablet_manager.cc:1398] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:16.830150 15657 tablet_bootstrap.cc:654] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.836918 15655 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:16.836984 15657 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:16.837106 15655 ts_tablet_manager.cc:1367] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.003s
I20240503 08:14:16.837152 15657 ts_tablet_manager.cc:1367] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.002s
I20240503 08:14:16.837492 15660 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:16.837657 15660 ts_tablet_manager.cc:1367] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.000s
I20240503 08:14:16.837855 15657 raft_consensus.cc:348] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:16.837931 15655 raft_consensus.cc:348] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.838058 15657 raft_consensus.cc:374] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.838124 15655 raft_consensus.cc:374] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.838135 15657 raft_consensus.cc:729] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.838259 15655 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.838469 15660 raft_consensus.cc:348] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.838413 15657 consensus_queue.cc:260] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:16.838488 15655 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.838680 15660 raft_consensus.cc:374] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.838776 15660 raft_consensus.cc:729] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.839031 15660 consensus_queue.cc:260] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.839676 15660 ts_tablet_manager.cc:1398] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:16.839851 15657 ts_tablet_manager.cc:1398] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:16.839948 15660 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:16.840400 15655 ts_tablet_manager.cc:1398] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:14:16.840678 15655 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:16.841558 15660 tablet_bootstrap.cc:654] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.843892 15655 tablet_bootstrap.cc:654] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.846575 15655 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:16.846621 15660 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:16.846777 15660 ts_tablet_manager.cc:1367] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 08:14:16.846777 15655 ts_tablet_manager.cc:1367] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.001s
I20240503 08:14:16.847553 15655 raft_consensus.cc:348] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.847553 15660 raft_consensus.cc:348] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:16.847790 15660 raft_consensus.cc:374] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.847790 15655 raft_consensus.cc:374] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.847875 15660 raft_consensus.cc:729] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.847880 15655 raft_consensus.cc:729] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.848163 15660 consensus_queue.cc:260] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:16.848134 15655 consensus_queue.cc:260] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.848695 15660 ts_tablet_manager.cc:1398] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:14:16.849030 15660 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:16.850600 15655 ts_tablet_manager.cc:1398] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:14:16.850630 15660 tablet_bootstrap.cc:654] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.850936 15655 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:16.852907 15655 tablet_bootstrap.cc:654] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.854472 15660 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:16.854624 15660 ts_tablet_manager.cc:1367] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.002s
I20240503 08:14:16.854939 15655 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:16.855077 15655 ts_tablet_manager.cc:1367] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.001s
I20240503 08:14:16.855419 15660 raft_consensus.cc:348] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.855631 15660 raft_consensus.cc:374] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.855737 15660 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.855854 15655 raft_consensus.cc:348] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:16.856050 15655 raft_consensus.cc:374] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.855970 15660 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.856141 15655 raft_consensus.cc:729] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.856370 15655 consensus_queue.cc:260] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:16.856559 15660 ts_tablet_manager.cc:1398] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:14:16.856895 15655 ts_tablet_manager.cc:1398] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:14:16.941794 15591 raft_consensus.cc:482] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:16.942031 15591 raft_consensus.cc:504] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.942958 15591 leader_election.cc:288] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:16.943828 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12df78f752c34fe3885b235c4a19e325" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:16.944069 14750 raft_consensus.cc:2454] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:16.944892 14817 leader_election.cc:302] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:16.945185 15591 raft_consensus.cc:2781] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:16.945302 15591 raft_consensus.cc:482] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:16.945430 15591 raft_consensus.cc:3037] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:16.948154 15591 raft_consensus.cc:504] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.948761 15591 leader_election.cc:288] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:16.949306 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12df78f752c34fe3885b235c4a19e325" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:16.949589 14750 raft_consensus.cc:3037] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:16.951442 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12df78f752c34fe3885b235c4a19e325" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:16.951442 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12df78f752c34fe3885b235c4a19e325" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:16.951761 14490 raft_consensus.cc:2454] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:16.952086 14750 raft_consensus.cc:2454] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:16.952500 14817 leader_election.cc:302] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:14:16.952836 15591 raft_consensus.cc:2781] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:16.952998 15591 raft_consensus.cc:686] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:14:16.953281 15591 consensus_queue.cc:237] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:16.956247 14343 catalog_manager.cc:5484] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:17.029956 15554 raft_consensus.cc:482] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:17.030148 15554 raft_consensus.cc:504] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.030929 15554 leader_election.cc:288] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:17.031546 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b13cdf4b234f17872f5d1588915968" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:17.031564 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b13cdf4b234f17872f5d1588915968" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:17.031824 14879 raft_consensus.cc:2454] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:17.031829 14620 raft_consensus.cc:2454] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:17.032310 14423 leader_election.cc:302] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:17.032588 15554 raft_consensus.cc:2781] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:17.032716 15554 raft_consensus.cc:482] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:17.032821 15554 raft_consensus.cc:3037] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.034962 15554 raft_consensus.cc:504] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.035619 15554 leader_election.cc:288] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:17.036175 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b13cdf4b234f17872f5d1588915968" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:17.036170 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b13cdf4b234f17872f5d1588915968" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:17.036438 14620 raft_consensus.cc:3037] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.036438 14879 raft_consensus.cc:3037] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.039368 14879 raft_consensus.cc:2454] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.039368 14620 raft_consensus.cc:2454] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.039904 14424 leader_election.cc:302] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:17.040247 15554 raft_consensus.cc:2781] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:17.040638 15554 raft_consensus.cc:686] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:17.040953 15554 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.044607 14343 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:14:17.049015 15671 raft_consensus.cc:482] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:17.049208 15671 raft_consensus.cc:504] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.049852 15671 leader_election.cc:288] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:17.050315 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:17.050540 14619 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:17.050590 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:17.050837 14879 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:17.050936 14423 leader_election.cc:302] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:17.051294 15554 raft_consensus.cc:2781] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:17.051422 15554 raft_consensus.cc:482] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:17.051520 15554 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.053550 15554 raft_consensus.cc:504] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.054128 15554 leader_election.cc:288] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:17.054601 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:17.054602 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:17.054797 14879 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.054797 14620 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.057608 14879 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.057610 14620 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.057994 14424 leader_election.cc:302] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:17.058287 15554 raft_consensus.cc:2781] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:17.058408 15554 raft_consensus.cc:686] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:17.058696 15554 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.061206 14343 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:17.377020 15591 consensus_queue.cc:1035] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:17.386380 14490 raft_consensus.cc:3037] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.391252 15512 consensus_queue.cc:1035] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:17.481911 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:17.482354 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:14:17.483201 15554 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:17.488459 15671 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:17.492647 15674 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:14:17.601316 15554 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:17.611927 15671 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 220.056 ms
  time per row: 0.110028 ms
I20240503 08:14:17.856756 14344 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:17.866039 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.866572 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.866528 14469 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.866938 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.867635 14730 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.867895 14860 tablet_service.cc:1467] Processing CreateTablet for tablet a87c5af03f4d47d38bbfbd225c7195d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.868053 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.868283 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c5af03f4d47d38bbfbd225c7195d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.874629 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 0780fb979eee4bae98a014d9721c9441 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.875105 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0780fb979eee4bae98a014d9721c9441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.875360 14729 tablet_service.cc:1467] Processing CreateTablet for tablet a87c5af03f4d47d38bbfbd225c7195d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.875735 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c5af03f4d47d38bbfbd225c7195d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.878129 15739 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:17.878465 15738 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:17.880076 15739 tablet_bootstrap.cc:654] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.880076 15738 tablet_bootstrap.cc:654] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.881220 14731 tablet_service.cc:1467] Processing CreateTablet for tablet 0780fb979eee4bae98a014d9721c9441 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.881662 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0780fb979eee4bae98a014d9721c9441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.881950 14469 tablet_service.cc:1467] Processing CreateTablet for tablet a87c5af03f4d47d38bbfbd225c7195d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.882411 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c5af03f4d47d38bbfbd225c7195d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.887168 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 0780fb979eee4bae98a014d9721c9441 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:17.887728 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0780fb979eee4bae98a014d9721c9441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:17.890802 15742 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:17.891501 15743 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:17.893307 15743 tablet_bootstrap.cc:654] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.894301 15742 tablet_bootstrap.cc:654] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.898591 15739 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:17.898809 15739 ts_tablet_manager.cc:1367] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.006s
I20240503 08:14:17.899941 15739 raft_consensus.cc:348] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.900233 15739 raft_consensus.cc:374] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.900315 15739 raft_consensus.cc:729] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.901872 15743 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:17.900626 15739 consensus_queue.cc:260] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.902037 15743 ts_tablet_manager.cc:1367] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.009s
I20240503 08:14:17.902853 15739 ts_tablet_manager.cc:1398] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:14:17.903154 15742 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:17.903198 15743 raft_consensus.cc:348] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.903414 15743 raft_consensus.cc:374] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.903371 15742 ts_tablet_manager.cc:1367] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.008s
I20240503 08:14:17.903507 15743 raft_consensus.cc:729] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.903787 15743 consensus_queue.cc:260] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.904501 15742 raft_consensus.cc:348] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.904785 15742 raft_consensus.cc:374] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.904927 15742 raft_consensus.cc:729] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.905144 15739 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:17.905328 15742 consensus_queue.cc:260] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.907163 15738 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:17.907339 15738 ts_tablet_manager.cc:1367] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.009s
I20240503 08:14:17.908133 15738 raft_consensus.cc:348] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.908361 15738 raft_consensus.cc:374] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.908447 15738 raft_consensus.cc:729] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.908660 15738 consensus_queue.cc:260] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.909186 15738 ts_tablet_manager.cc:1398] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:17.909487 15738 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:17.911223 15739 tablet_bootstrap.cc:654] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.917822 15738 tablet_bootstrap.cc:654] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.919687 15742 ts_tablet_manager.cc:1398] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.002s
I20240503 08:14:17.922654 15742 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:17.922739 15739 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:17.922912 15739 ts_tablet_manager.cc:1367] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.005s
I20240503 08:14:17.923799 15739 raft_consensus.cc:348] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.924057 15739 raft_consensus.cc:374] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.924149 15739 raft_consensus.cc:729] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.924443 15739 consensus_queue.cc:260] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.924949 15742 tablet_bootstrap.cc:654] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.924981 15554 raft_consensus.cc:482] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:17.925158 15554 raft_consensus.cc:504] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.925850 15554 leader_election.cc:288] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:17.926519 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc0fe7d81e149d5b82fabfdc11d66b4" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:17.926692 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc0fe7d81e149d5b82fabfdc11d66b4" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:17.926903 15739 ts_tablet_manager.cc:1398] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 08:14:17.926929 14620 raft_consensus.cc:2454] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
W20240503 08:14:17.927098 14426 leader_election.cc:341] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:17.927350 14423 leader_election.cc:302] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:14:17.928202 15738 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:17.928372 15738 ts_tablet_manager.cc:1367] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.019s	user 0.001s	sys 0.003s
I20240503 08:14:17.929651 15733 raft_consensus.cc:2781] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:17.929756 15733 raft_consensus.cc:482] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:17.929857 15733 raft_consensus.cc:3037] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.930660 15738 raft_consensus.cc:348] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.930907 15738 raft_consensus.cc:374] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.931005 15738 raft_consensus.cc:729] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.931290 15738 consensus_queue.cc:260] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.931946 15738 ts_tablet_manager.cc:1398] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:14:17.933063 15733 raft_consensus.cc:504] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.934587 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc0fe7d81e149d5b82fabfdc11d66b4" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:17.934796 14620 raft_consensus.cc:3037] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.937638 14620 raft_consensus.cc:2454] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.938076 14423 leader_election.cc:302] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:17.938437 15554 raft_consensus.cc:2781] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:17.938602 15554 raft_consensus.cc:686] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:17.938874 15554 consensus_queue.cc:237] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.939592 15742 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:17.940680 15733 leader_election.cc:288] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:17.940740 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc0fe7d81e149d5b82fabfdc11d66b4" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:17.940865 15742 ts_tablet_manager.cc:1367] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.018s	user 0.000s	sys 0.006s
W20240503 08:14:17.941334 14426 leader_election.cc:341] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:17.941668 15742 raft_consensus.cc:348] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.941857 15742 raft_consensus.cc:374] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.941951 15742 raft_consensus.cc:729] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.942268 15742 consensus_queue.cc:260] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.943493 15742 ts_tablet_manager.cc:1398] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:17.946540 15743 ts_tablet_manager.cc:1398] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.044s	user 0.012s	sys 0.011s
I20240503 08:14:17.946897 15743 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:17.948566 15743 tablet_bootstrap.cc:654] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.950500 15743 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:17.950711 15743 ts_tablet_manager.cc:1367] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.002s
I20240503 08:14:17.951857 15743 raft_consensus.cc:348] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.952044 15743 raft_consensus.cc:374] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.952137 15743 raft_consensus.cc:729] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.952139 14344 catalog_manager.cc:5484] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:17.952390 15743 consensus_queue.cc:260] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:17.952811 15743 ts_tablet_manager.cc:1398] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:14:17.953111 15743 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:17.954823 15554 raft_consensus.cc:482] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:17.954986 15554 raft_consensus.cc:504] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.955477 15743 tablet_bootstrap.cc:654] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:17.955678 15554 leader_election.cc:288] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:17.956319 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c5af03f4d47d38bbfbd225c7195d1" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:17.956634 14750 raft_consensus.cc:2454] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:17.956707 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c5af03f4d47d38bbfbd225c7195d1" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:17.956964 14879 raft_consensus.cc:2454] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:17.957194 14426 leader_election.cc:302] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:17.957594 15554 raft_consensus.cc:2781] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:17.957721 15554 raft_consensus.cc:482] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:17.957856 15554 raft_consensus.cc:3037] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.958911 15743 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:17.959069 15743 ts_tablet_manager.cc:1367] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.001s
I20240503 08:14:17.959849 15743 raft_consensus.cc:348] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.960050 15743 raft_consensus.cc:374] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:17.960144 15743 raft_consensus.cc:729] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:17.960366 15743 consensus_queue.cc:260] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:17.960764 15554 raft_consensus.cc:504] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.960902 15743 ts_tablet_manager.cc:1398] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:14:17.961580 15554 leader_election.cc:288] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:17.961812 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c5af03f4d47d38bbfbd225c7195d1" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:17.961839 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87c5af03f4d47d38bbfbd225c7195d1" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:17.961956 14750 raft_consensus.cc:3037] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.962061 14879 raft_consensus.cc:3037] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:17.964165 14750 raft_consensus.cc:2454] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.964534 14426 leader_election.cc:302] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:17.964622 14879 raft_consensus.cc:2454] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:17.964810 15554 raft_consensus.cc:2781] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:17.965018 15554 raft_consensus.cc:686] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:17.965323 15554 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:17.967942 14344 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:14:18.078716 15677 raft_consensus.cc:482] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:18.078908 15677 raft_consensus.cc:504] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:18.079644 15677 leader_election.cc:288] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:18.080267 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0780fb979eee4bae98a014d9721c9441" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:18.080284 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0780fb979eee4bae98a014d9721c9441" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:18.080487 14750 raft_consensus.cc:2454] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:18.080569 14620 raft_consensus.cc:2454] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:18.080952 14817 leader_election.cc:302] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:18.081254 15677 raft_consensus.cc:2781] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:18.081363 15677 raft_consensus.cc:482] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:18.081483 15677 raft_consensus.cc:3037] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:18.083508 15677 raft_consensus.cc:504] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:18.084043 15677 leader_election.cc:288] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:18.084615 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0780fb979eee4bae98a014d9721c9441" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:18.084615 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0780fb979eee4bae98a014d9721c9441" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:18.084833 14620 raft_consensus.cc:3037] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:18.084859 14750 raft_consensus.cc:3037] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:18.087762 14750 raft_consensus.cc:2454] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:18.087759 14620 raft_consensus.cc:2454] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:18.088219 14814 leader_election.cc:302] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:18.088531 15677 raft_consensus.cc:2781] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:18.088685 15677 raft_consensus.cc:686] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:14:18.088999 15677 consensus_queue.cc:237] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:18.091766 14344 catalog_manager.cc:5484] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:14:18.331480 15554 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:18.336571 14750 raft_consensus.cc:3037] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:18.339447 15554 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:18.471025 15677 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:18.477408 15512 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:18.511314 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:18.511677 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:18.519395 15755 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:18.523511 15554 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:18.528235 15554 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 229.69 ms
  time per row: 0.114845 ms
I20240503 08:14:18.901878 14347 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:18.911545 14469 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6c920958214148b3c3d5ac995e99e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.911779 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6c920958214148b3c3d5ac995e99e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.912052 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c920958214148b3c3d5ac995e99e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.912218 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c920958214148b3c3d5ac995e99e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.914139 14860 tablet_service.cc:1467] Processing CreateTablet for tablet b19df3ae9f2b44a3818855f259962fff (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.914597 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df3ae9f2b44a3818855f259962fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.914911 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 1c687520c0a349fd8438dacbd5011001 (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.915284 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c687520c0a349fd8438dacbd5011001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.915243 14599 tablet_service.cc:1467] Processing CreateTablet for tablet b19df3ae9f2b44a3818855f259962fff (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.915612 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df3ae9f2b44a3818855f259962fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.918620 14729 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6c920958214148b3c3d5ac995e99e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.919113 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c920958214148b3c3d5ac995e99e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.922757 14730 tablet_service.cc:1467] Processing CreateTablet for tablet 1c687520c0a349fd8438dacbd5011001 (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.923334 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c687520c0a349fd8438dacbd5011001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.926913 15820 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:18.927259 14470 tablet_service.cc:1467] Processing CreateTablet for tablet b19df3ae9f2b44a3818855f259962fff (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.927366 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 1c687520c0a349fd8438dacbd5011001 (DEFAULT_TABLE table=rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:18.927721 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df3ae9f2b44a3818855f259962fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.927945 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c687520c0a349fd8438dacbd5011001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:18.929203 15820 tablet_bootstrap.cc:654] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.937254 15820 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:18.937434 15820 ts_tablet_manager.cc:1367] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.005s
I20240503 08:14:18.938328 15820 raft_consensus.cc:348] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.938553 15820 raft_consensus.cc:374] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.938658 15820 raft_consensus.cc:729] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.938910 15820 consensus_queue.cc:260] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.939602 15820 ts_tablet_manager.cc:1398] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:14:18.941960 15821 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:18.943001 15823 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:18.946579 15821 tablet_bootstrap.cc:654] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.947767 15823 tablet_bootstrap.cc:654] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.949520 15821 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:18.949673 15821 ts_tablet_manager.cc:1367] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240503 08:14:18.950372 15821 raft_consensus.cc:348] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.950500 15820 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:18.950673 15821 raft_consensus.cc:374] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.950842 15821 raft_consensus.cc:729] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.951167 15821 consensus_queue.cc:260] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.952031 15820 tablet_bootstrap.cc:654] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.952814 15821 ts_tablet_manager.cc:1398] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:18.953194 15821 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:18.956887 15827 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:18.958057 15821 tablet_bootstrap.cc:654] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.960027 15827 tablet_bootstrap.cc:654] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.960367 15823 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:18.960551 15823 ts_tablet_manager.cc:1367] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.018s	user 0.003s	sys 0.000s
I20240503 08:14:18.961473 15823 raft_consensus.cc:348] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.961711 15823 raft_consensus.cc:374] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.961869 15823 raft_consensus.cc:729] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.963055 15823 consensus_queue.cc:260] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.965526 15823 ts_tablet_manager.cc:1398] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 08:14:18.965859 15823 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:18.970748 15823 tablet_bootstrap.cc:654] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.973034 15821 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:18.973217 15821 ts_tablet_manager.cc:1367] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.000s
I20240503 08:14:18.973974 15821 raft_consensus.cc:348] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.974232 15821 raft_consensus.cc:374] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.974334 15821 raft_consensus.cc:729] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.975152 15827 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:18.975306 15827 ts_tablet_manager.cc:1367] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.003s
I20240503 08:14:18.976030 15821 consensus_queue.cc:260] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.976668 15820 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:18.976822 15820 ts_tablet_manager.cc:1367] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.012s
I20240503 08:14:18.976838 15827 raft_consensus.cc:348] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.977039 15827 raft_consensus.cc:374] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.977133 15827 raft_consensus.cc:729] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.977387 15827 consensus_queue.cc:260] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.977488 15820 raft_consensus.cc:348] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.977707 15820 raft_consensus.cc:374] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.977785 15820 raft_consensus.cc:729] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.978016 15820 consensus_queue.cc:260] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.978592 15820 ts_tablet_manager.cc:1398] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:18.979738 15821 ts_tablet_manager.cc:1398] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240503 08:14:18.981861 15827 ts_tablet_manager.cc:1398] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240503 08:14:18.982208 15827 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:18.983975 15827 tablet_bootstrap.cc:654] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.984536 15823 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:18.984724 15823 ts_tablet_manager.cc:1367] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.000s
I20240503 08:14:18.985522 15823 raft_consensus.cc:348] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.985772 15823 raft_consensus.cc:374] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.985848 15823 raft_consensus.cc:729] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.986090 15823 consensus_queue.cc:260] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.987221 15827 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:18.987289 15823 ts_tablet_manager.cc:1398] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:14:18.987388 15827 ts_tablet_manager.cc:1367] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.001s
I20240503 08:14:18.987658 15823 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:18.988415 15827 raft_consensus.cc:348] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.988767 15746 raft_consensus.cc:482] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:18.988813 15827 raft_consensus.cc:374] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:18.988958 15827 raft_consensus.cc:729] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:18.988927 15746 raft_consensus.cc:504] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:18.989313 15827 consensus_queue.cc:260] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:18.989992 15746 leader_election.cc:288] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:18.991338 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c920958214148b3c3d5ac995e99e2" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:18.991575 14489 raft_consensus.cc:2454] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:18.992653 14687 leader_election.cc:302] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:18.994906 15823 tablet_bootstrap.cc:654] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:18.995276 15746 raft_consensus.cc:2781] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:18.995400 15746 raft_consensus.cc:482] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:18.995512 15746 raft_consensus.cc:3037] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:18.999725 15746 raft_consensus.cc:504] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:19.001359 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c920958214148b3c3d5ac995e99e2" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:19.001672 14489 raft_consensus.cc:3037] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.005805 14489 raft_consensus.cc:2454] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:19.006633 14687 leader_election.cc:302] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:19.007340 15746 leader_election.cc:288] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:19.009471 15746 raft_consensus.cc:2781] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:19.009693 15746 raft_consensus.cc:686] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:19.010299 15746 consensus_queue.cc:237] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:19.021718 15823 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:19.021909 15823 ts_tablet_manager.cc:1367] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.016s
I20240503 08:14:19.022699 15827 ts_tablet_manager.cc:1398] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.035s	user 0.011s	sys 0.022s
I20240503 08:14:19.022841 15823 raft_consensus.cc:348] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.023059 15823 raft_consensus.cc:374] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:19.023159 15823 raft_consensus.cc:729] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:19.022944 14349 catalog_manager.cc:5484] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:14:19.023416 15823 consensus_queue.cc:260] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.023878 15823 ts_tablet_manager.cc:1398] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:14:19.027873 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c920958214148b3c3d5ac995e99e2" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:19.028177 14620 raft_consensus.cc:2454] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:19.028518 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6c920958214148b3c3d5ac995e99e2" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:19.028695 14619 raft_consensus.cc:3037] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.030757 14619 raft_consensus.cc:2454] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:19.154886 15810 raft_consensus.cc:482] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:19.155083 15810 raft_consensus.cc:504] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.155834 15810 leader_election.cc:288] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:19.156531 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c687520c0a349fd8438dacbd5011001" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:19.156544 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c687520c0a349fd8438dacbd5011001" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:19.156816 14879 raft_consensus.cc:2454] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:19.156817 14749 raft_consensus.cc:2454] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:19.157382 14426 leader_election.cc:302] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:19.157716 15810 raft_consensus.cc:2781] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:19.157842 15810 raft_consensus.cc:482] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:19.157938 15810 raft_consensus.cc:3037] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.160143 15810 raft_consensus.cc:504] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.160703 15810 leader_election.cc:288] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:19.161284 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c687520c0a349fd8438dacbd5011001" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:19.161284 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c687520c0a349fd8438dacbd5011001" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:19.161473 14879 raft_consensus.cc:3037] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.161477 14749 raft_consensus.cc:3037] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.163550 14879 raft_consensus.cc:2454] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:19.163573 14749 raft_consensus.cc:2454] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:19.164045 14424 leader_election.cc:302] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:19.164386 15810 raft_consensus.cc:2781] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:19.164513 15810 raft_consensus.cc:686] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:19.164794 15810 consensus_queue.cc:237] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.167668 14347 catalog_manager.cc:5484] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:19.255812 15480 raft_consensus.cc:482] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:19.256002 15480 raft_consensus.cc:504] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.256645 15480 leader_election.cc:288] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:19.257078 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df3ae9f2b44a3818855f259962fff" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:19.257138 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df3ae9f2b44a3818855f259962fff" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:19.257297 14879 raft_consensus.cc:2454] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:19.257375 14489 raft_consensus.cc:2454] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:19.257690 14555 leader_election.cc:302] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:19.257954 15480 raft_consensus.cc:2781] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:19.258071 15480 raft_consensus.cc:482] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:19.258194 15480 raft_consensus.cc:3037] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.260349 15480 raft_consensus.cc:504] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.260906 15480 leader_election.cc:288] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:19.261265 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df3ae9f2b44a3818855f259962fff" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:19.261350 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df3ae9f2b44a3818855f259962fff" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:19.261441 14879 raft_consensus.cc:3037] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.261544 14489 raft_consensus.cc:3037] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:19.263303 14879 raft_consensus.cc:2454] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:19.263525 14489 raft_consensus.cc:2454] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:19.263698 14555 leader_election.cc:302] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:19.263974 15480 raft_consensus.cc:2781] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:19.264305 15480 raft_consensus.cc:686] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:19.264566 15480 consensus_queue.cc:237] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:19.267192 14347 catalog_manager.cc:5484] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:19.511461 15832 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:19.519004 15746 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:19.595453 15810 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:19.601584 15807 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:19.676093 15840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:19.676492 15840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:19.684523 15840 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:14:19.777695 15480 consensus_queue.cc:1035] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:19.784243 15480 consensus_queue.cc:1035] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 247.729 ms
  time per row: 0.123865 ms
I20240503 08:14:20.083976 14343 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:20.092916 14729 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa1031b9e30478597da5ad3708bdff5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.093108 14599 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa1031b9e30478597da5ad3708bdff5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.093413 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa1031b9e30478597da5ad3708bdff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.093526 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa1031b9e30478597da5ad3708bdff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.093722 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa1031b9e30478597da5ad3708bdff5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.094146 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa1031b9e30478597da5ad3708bdff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.095005 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 5c595bb19d6946b0b1f853064892e946 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.095404 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c595bb19d6946b0b1f853064892e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.099048 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 562e1ec5c34945298cf457758328057f (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.099475 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 562e1ec5c34945298cf457758328057f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.099534 14730 tablet_service.cc:1467] Processing CreateTablet for tablet 562e1ec5c34945298cf457758328057f (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.099908 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 562e1ec5c34945298cf457758328057f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.102572 14731 tablet_service.cc:1467] Processing CreateTablet for tablet 5c595bb19d6946b0b1f853064892e946 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.102978 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c595bb19d6946b0b1f853064892e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.107203 14601 tablet_service.cc:1467] Processing CreateTablet for tablet 5c595bb19d6946b0b1f853064892e946 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.107677 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c595bb19d6946b0b1f853064892e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.108641 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 562e1ec5c34945298cf457758328057f (DEFAULT_TABLE table=rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:20.109009 15912 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:20.109112 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 562e1ec5c34945298cf457758328057f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:20.112068 15912 tablet_bootstrap.cc:654] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.121196 15912 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:20.121520 15912 ts_tablet_manager.cc:1367] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.000s
I20240503 08:14:20.122602 15912 raft_consensus.cc:348] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.123003 15912 raft_consensus.cc:374] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.123188 15912 raft_consensus.cc:729] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.123555 15912 consensus_queue.cc:260] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.125355 15912 ts_tablet_manager.cc:1398] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:14:20.125751 15871 raft_consensus.cc:482] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:20.125751 15912 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:20.125943 15871 raft_consensus.cc:504] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.126700 15871 leader_election.cc:288] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:20.127269 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa1031b9e30478597da5ad3708bdff5" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:20.127761 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa1031b9e30478597da5ad3708bdff5" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
W20240503 08:14:20.128070 14817 leader_election.cc:341] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:14:20.128737 14814 leader_election.cc:341] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:20.128901 14814 leader_election.cc:302] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938; no voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:14:20.129493 15871 raft_consensus.cc:2726] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:20.133379 15912 tablet_bootstrap.cc:654] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.135416 15915 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:20.137465 15914 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:20.138756 15911 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:20.139235 15914 tablet_bootstrap.cc:654] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.139294 15915 tablet_bootstrap.cc:654] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.140161 15912 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:20.140352 15912 ts_tablet_manager.cc:1367] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I20240503 08:14:20.140921 15911 tablet_bootstrap.cc:654] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.147917 15911 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:20.148092 15911 ts_tablet_manager.cc:1367] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240503 08:14:20.148420 15914 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:20.148561 15914 ts_tablet_manager.cc:1367] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.004s
I20240503 08:14:20.149394 15914 raft_consensus.cc:348] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.149600 15914 raft_consensus.cc:374] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.149696 15914 raft_consensus.cc:729] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.149928 15914 consensus_queue.cc:260] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.150564 15914 ts_tablet_manager.cc:1398] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:20.151008 15914 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:20.150471 15911 raft_consensus.cc:348] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.151093 15911 raft_consensus.cc:374] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.151192 15911 raft_consensus.cc:729] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.151445 15911 consensus_queue.cc:260] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.152109 15911 ts_tablet_manager.cc:1398] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:14:20.152416 15911 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:20.152647 15914 tablet_bootstrap.cc:654] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.154227 15911 tablet_bootstrap.cc:654] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.158952 15912 raft_consensus.cc:348] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.159404 15912 raft_consensus.cc:374] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.159554 15912 raft_consensus.cc:729] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.159938 15912 consensus_queue.cc:260] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.160750 15912 ts_tablet_manager.cc:1398] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.000s
I20240503 08:14:20.166098 15911 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:20.166370 15911 ts_tablet_manager.cc:1367] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.005s
I20240503 08:14:20.167414 15911 raft_consensus.cc:348] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.167728 15911 raft_consensus.cc:374] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.167829 15911 raft_consensus.cc:729] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.168184 15911 consensus_queue.cc:260] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.168998 15911 ts_tablet_manager.cc:1398] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:20.169346 15911 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:20.172541 15915 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:20.172600 15911 tablet_bootstrap.cc:654] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.172716 15914 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:20.172900 15914 ts_tablet_manager.cc:1367] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.008s
I20240503 08:14:20.172991 15915 ts_tablet_manager.cc:1367] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.038s	user 0.017s	sys 0.017s
I20240503 08:14:20.173686 15914 raft_consensus.cc:348] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.173988 15914 raft_consensus.cc:374] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.174125 15914 raft_consensus.cc:729] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.173981 15915 raft_consensus.cc:348] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.174343 15915 raft_consensus.cc:374] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.174494 15915 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.174569 15914 consensus_queue.cc:260] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.174753 15915 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.175361 15914 ts_tablet_manager.cc:1398] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:14:20.175621 15914 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:20.175961 15915 ts_tablet_manager.cc:1398] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:20.176054 15911 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:20.176189 15911 ts_tablet_manager.cc:1367] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.002s
I20240503 08:14:20.176775 15911 raft_consensus.cc:348] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.176924 15911 raft_consensus.cc:374] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.177014 15911 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.177301 15914 tablet_bootstrap.cc:654] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:20.177237 15911 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.177814 15911 ts_tablet_manager.cc:1398] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:14:20.179885 15914 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:20.180069 15914 ts_tablet_manager.cc:1367] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240503 08:14:20.180994 15914 raft_consensus.cc:348] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.181218 15914 raft_consensus.cc:374] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:20.181324 15914 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:20.181545 15914 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.182148 15914 ts_tablet_manager.cc:1398] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 08:14:20.189329 15898 raft_consensus.cc:482] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:20.189497 15898 raft_consensus.cc:504] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.190199 15898 leader_election.cc:288] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:20.190562 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "562e1ec5c34945298cf457758328057f" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:20.190667 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "562e1ec5c34945298cf457758328057f" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:20.190855 14879 raft_consensus.cc:2454] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:20.190879 14751 raft_consensus.cc:2454] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:20.191325 14557 leader_election.cc:302] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:20.191601 15898 raft_consensus.cc:2781] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:20.191725 15898 raft_consensus.cc:482] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:20.191818 15898 raft_consensus.cc:3037] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.193662 15898 raft_consensus.cc:504] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.194200 15898 leader_election.cc:288] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:20.194615 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "562e1ec5c34945298cf457758328057f" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:20.194610 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "562e1ec5c34945298cf457758328057f" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:20.194777 14751 raft_consensus.cc:3037] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.194802 14879 raft_consensus.cc:3037] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.196586 14751 raft_consensus.cc:2454] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:20.196664 14879 raft_consensus.cc:2454] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:20.196962 14557 leader_election.cc:302] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:20.197314 15898 raft_consensus.cc:2781] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:20.197469 15898 raft_consensus.cc:686] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:20.197751 15898 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:20.200873 14343 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:14:20.240365 15898 raft_consensus.cc:482] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:20.240569 15898 raft_consensus.cc:504] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.241206 15898 leader_election.cc:288] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:20.241695 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:20.241739 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:20.241943 14751 raft_consensus.cc:2454] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:20.242031 14490 raft_consensus.cc:2454] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:20.242357 14557 leader_election.cc:302] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:20.242642 15898 raft_consensus.cc:2781] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:20.242728 15898 raft_consensus.cc:482] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:20.242842 15898 raft_consensus.cc:3037] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.264014 15898 raft_consensus.cc:504] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.264676 15898 leader_election.cc:288] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:20.265300 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:20.265327 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:20.265515 14490 raft_consensus.cc:3037] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.265522 14751 raft_consensus.cc:3037] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.267436 14490 raft_consensus.cc:2454] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:20.267541 14751 raft_consensus.cc:2454] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:20.267916 14557 leader_election.cc:302] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:20.268239 15898 raft_consensus.cc:2781] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:20.268405 15898 raft_consensus.cc:686] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:20.268683 15898 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:20.271404 14343 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:20.516556 15746 raft_consensus.cc:482] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:20.516767 15746 raft_consensus.cc:504] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.517421 15746 leader_election.cc:288] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:20.517926 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa1031b9e30478597da5ad3708bdff5" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:20.517933 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa1031b9e30478597da5ad3708bdff5" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:20.518203 14879 raft_consensus.cc:2454] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:20.518209 14619 raft_consensus.cc:2454] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:20.518636 14684 leader_election.cc:302] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:20.519027 15746 raft_consensus.cc:2781] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:20.519156 15746 raft_consensus.cc:482] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:20.519253 15746 raft_consensus.cc:3037] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.521221 15746 raft_consensus.cc:504] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.521744 15746 leader_election.cc:288] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:20.522145 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa1031b9e30478597da5ad3708bdff5" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:20.522264 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa1031b9e30478597da5ad3708bdff5" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:20.522357 14619 raft_consensus.cc:3037] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.522483 14879 raft_consensus.cc:3037] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:20.524156 14619 raft_consensus.cc:2454] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:20.524266 14879 raft_consensus.cc:2454] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:20.524549 14684 leader_election.cc:302] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:20.524823 15746 raft_consensus.cc:2781] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:20.525132 15746 raft_consensus.cc:686] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:20.525411 15746 consensus_queue.cc:237] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:20.528020 14343 catalog_manager.cc:5484] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:20.599486 15898 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:20.617653 15873 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:20.648011 15873 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:20.653244 15873 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:20.946655 15928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:20.946986 15928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:20.955610 15928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:20.959730 15746 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:20.964725 15746 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'

INSERT report
    rows total: 2000
    time total: 196.301 ms
  time per row: 0.0981506 ms
I20240503 08:14:21.312089 14343 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:21.321141 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 5d16c2981213464cbefa44b6ca509182 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.321652 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d16c2981213464cbefa44b6ca509182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.321789 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 5d16c2981213464cbefa44b6ca509182 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.322240 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d16c2981213464cbefa44b6ca509182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.322587 14599 tablet_service.cc:1467] Processing CreateTablet for tablet 5d16c2981213464cbefa44b6ca509182 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.323026 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d16c2981213464cbefa44b6ca509182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.326622 14729 tablet_service.cc:1467] Processing CreateTablet for tablet 897a701b880641b08816f2d1a254ef03 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.326607 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 76a216295a8a4feaa2d3326035879073 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.326622 14469 tablet_service.cc:1467] Processing CreateTablet for tablet 897a701b880641b08816f2d1a254ef03 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.327010 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a216295a8a4feaa2d3326035879073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.327016 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897a701b880641b08816f2d1a254ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.327016 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897a701b880641b08816f2d1a254ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.328305 14470 tablet_service.cc:1467] Processing CreateTablet for tablet 76a216295a8a4feaa2d3326035879073 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.328697 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a216295a8a4feaa2d3326035879073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.331568 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 897a701b880641b08816f2d1a254ef03 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.331987 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897a701b880641b08816f2d1a254ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.340366 15990 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:21.342037 15992 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:21.344190 15990 tablet_bootstrap.cc:654] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.345252 15992 tablet_bootstrap.cc:654] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.347242 15993 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:21.349956 15993 tablet_bootstrap.cc:654] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.356350 15992 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:21.356690 15992 ts_tablet_manager.cc:1367] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.004s
I20240503 08:14:21.356734 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 76a216295a8a4feaa2d3326035879073 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:21.357553 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a216295a8a4feaa2d3326035879073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:21.358018 15992 raft_consensus.cc:348] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.358564 15992 raft_consensus.cc:374] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.358745 15992 raft_consensus.cc:729] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.359237 15992 consensus_queue.cc:260] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.360332 15992 ts_tablet_manager.cc:1398] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:14:21.360862 15992 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:21.363309 15992 tablet_bootstrap.cc:654] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.382483 15991 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:21.383141 15992 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:21.383878 15993 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:21.384109 15993 ts_tablet_manager.cc:1367] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.037s	user 0.008s	sys 0.000s
I20240503 08:14:21.384929 15993 raft_consensus.cc:348] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.385223 15993 raft_consensus.cc:374] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.385353 15993 raft_consensus.cc:729] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.386092 15993 consensus_queue.cc:260] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.387573 15993 ts_tablet_manager.cc:1398] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:21.387924 15992 ts_tablet_manager.cc:1367] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.027s	user 0.006s	sys 0.000s
I20240503 08:14:21.388774 15992 raft_consensus.cc:348] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.389052 15992 raft_consensus.cc:374] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.389082 15991 tablet_bootstrap.cc:654] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.390034 15992 raft_consensus.cc:729] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.390401 15992 consensus_queue.cc:260] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.391101 15992 ts_tablet_manager.cc:1398] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 08:14:21.394356 15990 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:21.394578 15990 ts_tablet_manager.cc:1367] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.054s	user 0.029s	sys 0.010s
I20240503 08:14:21.395509 15990 raft_consensus.cc:348] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.395772 15990 raft_consensus.cc:374] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.395874 15990 raft_consensus.cc:729] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.396139 15990 consensus_queue.cc:260] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.396687 15990 ts_tablet_manager.cc:1398] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:21.397091 15990 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:21.398762 15990 tablet_bootstrap.cc:654] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.401217 15991 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:21.401364 15991 ts_tablet_manager.cc:1367] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.000s
I20240503 08:14:21.402181 15991 raft_consensus.cc:348] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.402456 15991 raft_consensus.cc:374] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.402575 15991 raft_consensus.cc:729] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.402868 15991 consensus_queue.cc:260] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.403551 15991 ts_tablet_manager.cc:1398] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:21.404048 15991 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:21.405645 15991 tablet_bootstrap.cc:654] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.407912 15991 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:21.408138 15991 ts_tablet_manager.cc:1367] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:14:21.408401 15990 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:21.408596 15990 ts_tablet_manager.cc:1367] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240503 08:14:21.409067 15991 raft_consensus.cc:348] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.409370 15991 raft_consensus.cc:374] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.409457 15991 raft_consensus.cc:729] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.409379 15990 raft_consensus.cc:348] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.409605 15990 raft_consensus.cc:374] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.409695 15990 raft_consensus.cc:729] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.409710 15991 consensus_queue.cc:260] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.409935 15990 consensus_queue.cc:260] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.410295 15991 ts_tablet_manager.cc:1398] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:21.410511 15990 ts_tablet_manager.cc:1398] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:14:21.410647 15991 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:21.410841 15990 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:21.412334 15991 tablet_bootstrap.cc:654] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.412560 15990 tablet_bootstrap.cc:654] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:21.415655 15991 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:21.415809 15991 ts_tablet_manager.cc:1367] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:14:21.416312 15990 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:21.416467 15990 ts_tablet_manager.cc:1367] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.001s
I20240503 08:14:21.416577 15991 raft_consensus.cc:348] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.416766 15991 raft_consensus.cc:374] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.416849 15991 raft_consensus.cc:729] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.417080 15991 consensus_queue.cc:260] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.417207 15990 raft_consensus.cc:348] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.417395 15990 raft_consensus.cc:374] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:21.417490 15990 raft_consensus.cc:729] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:21.417629 15991 ts_tablet_manager.cc:1398] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:14:21.417706 15990 consensus_queue.cc:260] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.418331 15990 ts_tablet_manager.cc:1398] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:14:21.452153 15983 raft_consensus.cc:482] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:21.452353 15983 raft_consensus.cc:504] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.453151 15983 leader_election.cc:288] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:21.453635 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897a701b880641b08816f2d1a254ef03" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:21.453650 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897a701b880641b08816f2d1a254ef03" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:21.453856 14881 raft_consensus.cc:2454] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:21.453882 14489 raft_consensus.cc:2454] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:21.454380 14685 leader_election.cc:302] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:21.454663 15983 raft_consensus.cc:2781] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:21.454780 15983 raft_consensus.cc:482] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:21.454881 15983 raft_consensus.cc:3037] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.456750 15983 raft_consensus.cc:504] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.457365 15983 leader_election.cc:288] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:21.457777 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897a701b880641b08816f2d1a254ef03" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:21.457860 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897a701b880641b08816f2d1a254ef03" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:21.457967 14881 raft_consensus.cc:3037] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.458084 14489 raft_consensus.cc:3037] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.460065 14881 raft_consensus.cc:2454] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:21.460073 14489 raft_consensus.cc:2454] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:21.460506 14687 leader_election.cc:302] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:21.460839 15983 raft_consensus.cc:2781] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:21.460996 15983 raft_consensus.cc:686] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:21.461325 15983 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.464505 14347 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:21.636160 15934 raft_consensus.cc:482] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:21.636375 15934 raft_consensus.cc:504] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.637156 15934 leader_election.cc:288] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:21.637722 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d16c2981213464cbefa44b6ca509182" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:21.637750 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d16c2981213464cbefa44b6ca509182" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:21.637971 14489 raft_consensus.cc:2454] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:21.637992 14619 raft_consensus.cc:2454] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:21.638427 14817 leader_election.cc:302] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:21.638775 15934 raft_consensus.cc:2781] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:21.638895 15934 raft_consensus.cc:482] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:21.638969 15934 raft_consensus.cc:3037] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.640905 15934 raft_consensus.cc:504] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.641530 15934 leader_election.cc:288] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:21.642191 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d16c2981213464cbefa44b6ca509182" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:21.642200 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d16c2981213464cbefa44b6ca509182" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:21.642416 14489 raft_consensus.cc:3037] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.642449 14619 raft_consensus.cc:3037] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.644466 14489 raft_consensus.cc:2454] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:21.644724 14619 raft_consensus.cc:2454] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:21.644939 14817 leader_election.cc:302] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:21.645265 15934 raft_consensus.cc:2781] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:21.645694 15934 raft_consensus.cc:686] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:14:21.646013 15934 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:21.649206 14347 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:21.660893 15968 raft_consensus.cc:482] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:21.661084 15968 raft_consensus.cc:504] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.661705 15968 leader_election.cc:288] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:21.662243 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a216295a8a4feaa2d3326035879073" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:21.662325 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a216295a8a4feaa2d3326035879073" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:21.662521 14489 raft_consensus.cc:2454] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:21.662542 14881 raft_consensus.cc:2454] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:21.662973 14555 leader_election.cc:302] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:21.663301 15968 raft_consensus.cc:2781] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:21.663430 15968 raft_consensus.cc:482] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:21.663539 15968 raft_consensus.cc:3037] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.665591 15968 raft_consensus.cc:504] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.666144 15968 leader_election.cc:288] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:21.666568 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a216295a8a4feaa2d3326035879073" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:21.666671 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a216295a8a4feaa2d3326035879073" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:21.666810 14489 raft_consensus.cc:3037] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.666909 14881 raft_consensus.cc:3037] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:21.669720 14881 raft_consensus.cc:2454] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:21.669720 14489 raft_consensus.cc:2454] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:21.670279 14555 leader_election.cc:302] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:21.670599 15968 raft_consensus.cc:2781] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:21.670759 15968 raft_consensus.cc:686] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:21.671062 15968 consensus_queue.cc:237] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:21.673748 14347 catalog_manager.cc:5484] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:21.903470 15983 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:21.922441 15926 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:22.090907 16007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:22.091243 16007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:22.098985 16007 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:14:22.161047 15934 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:22.186985 15968 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:22.227721 16038 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:22.233590 15968 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 387.098 ms
  time per row: 0.193549 ms
I20240503 08:14:22.638428 14344 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:22.648447 14729 tablet_service.cc:1467] Processing CreateTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.648535 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.648967 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.649004 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.649574 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.649649 14599 tablet_service.cc:1467] Processing CreateTablet for tablet f19b7ef418974b06bdcc7125e8cf732d (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.649971 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.650611 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19b7ef418974b06bdcc7125e8cf732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.653667 14731 tablet_service.cc:1467] Processing CreateTablet for tablet 72a949e76ab5477680a4f9912e9672e9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.655246 14469 tablet_service.cc:1467] Processing CreateTablet for tablet 72a949e76ab5477680a4f9912e9672e9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.655287 14861 tablet_service.cc:1467] Processing CreateTablet for tablet 72a949e76ab5477680a4f9912e9672e9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.655642 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.655719 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.654131 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.662693 14860 tablet_service.cc:1467] Processing CreateTablet for tablet f19b7ef418974b06bdcc7125e8cf732d (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.663321 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19b7ef418974b06bdcc7125e8cf732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.654560 14730 tablet_service.cc:1467] Processing CreateTablet for tablet f19b7ef418974b06bdcc7125e8cf732d (DEFAULT_TABLE table=rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.666718 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19b7ef418974b06bdcc7125e8cf732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.675244 16079 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:22.680467 16079 tablet_bootstrap.cc:654] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.684934 16082 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:22.687608 16082 tablet_bootstrap.cc:654] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.691440 16080 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:22.692902 16079 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:22.693167 16079 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.005s
I20240503 08:14:22.693540 16080 tablet_bootstrap.cc:654] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.693894 16082 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:22.694053 16082 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.002s
I20240503 08:14:22.694089 16079 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.694360 16079 raft_consensus.cc:374] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.694497 16079 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.694762 16078 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:22.694769 16079 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.694906 16082 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.695349 16082 raft_consensus.cc:374] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.695418 16079 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:22.695484 16082 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.695729 16079 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:22.695745 16082 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.696316 16082 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:22.696761 16082 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:22.697494 16079 tablet_bootstrap.cc:654] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.698428 16082 tablet_bootstrap.cc:654] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.703308 16078 tablet_bootstrap.cc:654] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.706637 16079 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:22.706884 16079 ts_tablet_manager.cc:1367] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.004s
I20240503 08:14:22.707003 16078 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:22.707159 16078 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240503 08:14:22.707808 16078 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.707994 16078 raft_consensus.cc:374] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.707967 16079 raft_consensus.cc:348] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.708312 16079 raft_consensus.cc:374] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.708412 16079 raft_consensus.cc:729] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.708844 16079 consensus_queue.cc:260] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.710999 16079 ts_tablet_manager.cc:1398] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:14:22.711138 16078 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.711508 16078 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.712548 16080 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:22.712713 16080 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.000s
I20240503 08:14:22.713587 16080 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.713793 16080 raft_consensus.cc:374] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.713893 16080 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.714120 16080 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.714740 16080 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:22.715080 16080 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:22.716804 16080 tablet_bootstrap.cc:654] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.718152 16078 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240503 08:14:22.718575 16078 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:22.719635 16080 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:22.719650 16082 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:22.719823 16082 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.014s
I20240503 08:14:22.719841 16080 ts_tablet_manager.cc:1367] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 08:14:22.720631 16078 tablet_bootstrap.cc:654] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.720592 16080 raft_consensus.cc:348] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.720654 16082 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.720863 16080 raft_consensus.cc:374] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.720933 16082 raft_consensus.cc:374] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.720976 16080 raft_consensus.cc:729] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.721091 16082 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.721282 16080 consensus_queue.cc:260] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.721374 16082 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.722117 16082 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:14:22.722285 16080 ts_tablet_manager.cc:1398] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:22.728039 16078 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:22.728241 16078 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.005s
I20240503 08:14:22.728904 16078 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.729090 16078 raft_consensus.cc:374] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.729177 16078 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.729404 16078 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.729943 16078 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:14:22.730355 16078 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:22.731573 16078 tablet_bootstrap.cc:654] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.734642 16078 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:22.734786 16078 ts_tablet_manager.cc:1367] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.002s
I20240503 08:14:22.735486 16078 raft_consensus.cc:348] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.735643 16078 raft_consensus.cc:374] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.735708 16078 raft_consensus.cc:729] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.735857 16078 consensus_queue.cc:260] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.736292 16078 ts_tablet_manager.cc:1398] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:14:22.767525 16043 raft_consensus.cc:482] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.767715 16043 raft_consensus.cc:504] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.768357 16043 leader_election.cc:288] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:22.768899 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:22.768921 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:22.769121 14749 raft_consensus.cc:2454] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:22.769125 14881 raft_consensus.cc:2454] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:22.769588 14426 leader_election.cc:302] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:22.769878 16043 raft_consensus.cc:2781] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:22.769977 16043 raft_consensus.cc:482] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:22.770067 16043 raft_consensus.cc:3037] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.772001 16043 raft_consensus.cc:504] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.772526 16043 leader_election.cc:288] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:22.772930 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:22.773030 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:22.773166 14749 raft_consensus.cc:3037] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.773232 14881 raft_consensus.cc:3037] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.775138 14881 raft_consensus.cc:2454] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:22.775259 14749 raft_consensus.cc:2454] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:22.775561 14424 leader_election.cc:302] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:22.775871 16043 raft_consensus.cc:2781] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:22.776264 16043 raft_consensus.cc:686] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:22.776531 16043 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:22.779544 14341 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:22.813500 15926 raft_consensus.cc:482] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.813725 15926 raft_consensus.cc:504] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.814361 15926 leader_election.cc:288] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:22.814862 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19b7ef418974b06bdcc7125e8cf732d" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:22.814882 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19b7ef418974b06bdcc7125e8cf732d" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:22.815071 14619 raft_consensus.cc:2454] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:22.815124 14881 raft_consensus.cc:2454] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:22.815521 14684 leader_election.cc:302] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:22.815825 15926 raft_consensus.cc:2781] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:22.815924 15926 raft_consensus.cc:482] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:22.816008 15926 raft_consensus.cc:3037] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.817863 15926 raft_consensus.cc:504] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.818467 15926 leader_election.cc:288] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:22.818910 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19b7ef418974b06bdcc7125e8cf732d" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:22.818957 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19b7ef418974b06bdcc7125e8cf732d" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:22.819075 14619 raft_consensus.cc:3037] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.819231 14881 raft_consensus.cc:3037] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.821039 14619 raft_consensus.cc:2454] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:22.821062 14881 raft_consensus.cc:2454] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:22.821476 14684 leader_election.cc:302] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:22.821746 15926 raft_consensus.cc:2781] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:22.821906 15926 raft_consensus.cc:686] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:22.822281 15926 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:22.825233 14341 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:23.072548 16043 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:23.072814 16043 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:23.073473 16043 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:23.074048 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:23.074051 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:23.074283 14749 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:23.074342 14619 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:23.074674 14426 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:23.074985 16043 raft_consensus.cc:2781] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:23.075070 16043 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:23.075176 16043 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.077047 16043 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:23.077608 16043 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:23.078109 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:23.078190 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:23.078352 14749 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.078379 14619 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.081046 14619 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:23.081063 14749 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:23.081465 14423 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:23.081780 16043 raft_consensus.cc:2781] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:23.081949 16043 raft_consensus.cc:686] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:23.082276 16043 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:23.085211 14341 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:23.275535 15926 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.282487 16011 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.295635 16043 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.306136 16043 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:23.504021 16095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:23.504395 16095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:23.513005 16095 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:14:23.602638 16091 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.607539 16043 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:23.729667 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 69278.5us

INSERT report
    rows total: 2000
    time total: 273.017 ms
  time per row: 0.136508 ms
I20240503 08:14:23.960985 14348 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:23.972774 14729 tablet_service.cc:1467] Processing CreateTablet for tablet f7a5221747794a13aa4ed763006ba291 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:23.973098 14469 tablet_service.cc:1467] Processing CreateTablet for tablet f7a5221747794a13aa4ed763006ba291 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:23.973237 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a5221747794a13aa4ed763006ba291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:23.973495 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a5221747794a13aa4ed763006ba291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:23.976373 14468 tablet_service.cc:1467] Processing CreateTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:23.976788 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:23.978607 14861 tablet_service.cc:1467] Processing CreateTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:23.979041 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:23.983112 14600 tablet_service.cc:1467] Processing CreateTablet for tablet f7a5221747794a13aa4ed763006ba291 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:23.983356 14599 tablet_service.cc:1467] Processing CreateTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:23.983546 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a5221747794a13aa4ed763006ba291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:23.983855 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:23.991497 16164 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:23.993710 16164 tablet_bootstrap.cc:654] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:23.998807 16165 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:24.000497 14730 tablet_service.cc:1467] Processing CreateTablet for tablet c130eb7ea8a74f4b8d9b79e23af70067 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:24.001096 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130eb7ea8a74f4b8d9b79e23af70067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:24.003818 16165 tablet_bootstrap.cc:654] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.003770 14860 tablet_service.cc:1467] Processing CreateTablet for tablet c130eb7ea8a74f4b8d9b79e23af70067 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:24.003998 16166 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:24.004355 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130eb7ea8a74f4b8d9b79e23af70067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:24.006278 16166 tablet_bootstrap.cc:654] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.012503 14469 tablet_service.cc:1467] Processing CreateTablet for tablet c130eb7ea8a74f4b8d9b79e23af70067 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:24.013029 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130eb7ea8a74f4b8d9b79e23af70067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:24.013659 16165 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:24.013825 16165 ts_tablet_manager.cc:1367] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I20240503 08:14:24.014834 16165 raft_consensus.cc:348] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.015074 16165 raft_consensus.cc:374] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.015165 16165 raft_consensus.cc:729] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.015528 16165 consensus_queue.cc:260] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.018208 16165 ts_tablet_manager.cc:1398] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:14:24.018613 16165 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:24.020697 16165 tablet_bootstrap.cc:654] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.021896 16166 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:24.022176 16166 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.019s	user 0.000s	sys 0.005s
I20240503 08:14:24.023250 16166 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.023603 16166 raft_consensus.cc:374] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.023783 16166 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.024168 16166 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.025166 16166 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:24.033946 16165 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:24.034276 16165 ts_tablet_manager.cc:1367] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.004s
I20240503 08:14:24.035274 16164 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:24.035609 16164 ts_tablet_manager.cc:1367] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.044s	user 0.000s	sys 0.015s
I20240503 08:14:24.036876 16165 raft_consensus.cc:348] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.037133 16170 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:24.037254 16165 raft_consensus.cc:374] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.037398 16165 raft_consensus.cc:729] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.037794 16165 consensus_queue.cc:260] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.038333 16164 raft_consensus.cc:348] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.038703 16165 ts_tablet_manager.cc:1398] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:24.039778 16164 raft_consensus.cc:374] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.040100 16164 raft_consensus.cc:729] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.040423 16164 consensus_queue.cc:260] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.041278 16164 ts_tablet_manager.cc:1398] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 08:14:24.041360 16166 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:24.042517 16164 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:24.043571 16166 tablet_bootstrap.cc:654] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.044477 16164 tablet_bootstrap.cc:654] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.050894 16170 tablet_bootstrap.cc:654] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.060264 16170 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:24.060519 16170 ts_tablet_manager.cc:1367] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.000s
I20240503 08:14:24.060482 16128 raft_consensus.cc:482] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:24.060671 16128 raft_consensus.cc:504] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.061678 16128 leader_election.cc:288] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:24.062283 16166 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:24.062413 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f165b4a5647a48ad7ff951fb3fbf6" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:24.062507 16166 ts_tablet_manager.cc:1367] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.011s
I20240503 08:14:24.062587 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f165b4a5647a48ad7ff951fb3fbf6" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
W20240503 08:14:24.063282 14557 leader_election.cc:341] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:14:24.063558 14555 leader_election.cc:341] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:24.063453 16166 raft_consensus.cc:348] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.063670 14555 leader_election.cc:302] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288; no voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:14:24.063730 16166 raft_consensus.cc:374] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.063814 16166 raft_consensus.cc:729] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.063931 16128 raft_consensus.cc:2726] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:24.064142 16166 consensus_queue.cc:260] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.064589 16170 raft_consensus.cc:348] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.064860 16170 raft_consensus.cc:374] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.064901 16166 ts_tablet_manager.cc:1398] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:24.064972 16170 raft_consensus.cc:729] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.065199 16170 consensus_queue.cc:260] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.065701 16170 ts_tablet_manager.cc:1398] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240503 08:14:24.065994 16170 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:24.067796 16170 tablet_bootstrap.cc:654] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.068650 16164 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:24.068833 16164 ts_tablet_manager.cc:1367] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.003s
I20240503 08:14:24.069711 16164 raft_consensus.cc:348] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.069933 16164 raft_consensus.cc:374] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.070014 16164 raft_consensus.cc:729] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.070286 16164 consensus_queue.cc:260] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.072662 16164 ts_tablet_manager.cc:1398] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:14:24.072973 16164 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:24.074518 16164 tablet_bootstrap.cc:654] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:24.075130 16170 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:24.075335 16170 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240503 08:14:24.077540 16170 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.077900 16170 raft_consensus.cc:374] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.078011 16170 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.078403 16170 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.079282 16170 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:14:24.084339 16164 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:24.084525 16164 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.005s
I20240503 08:14:24.085348 16164 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.085515 16164 raft_consensus.cc:374] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:24.085582 16164 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:24.085808 16164 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.086320 16164 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240503 08:14:24.261251 16177 raft_consensus.cc:482] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:24.261452 16177 raft_consensus.cc:504] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.262250 16177 leader_election.cc:288] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:24.262789 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a5221747794a13aa4ed763006ba291" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:24.262874 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a5221747794a13aa4ed763006ba291" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:24.263039 14489 raft_consensus.cc:2454] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:24.263204 14749 raft_consensus.cc:2454] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 0.
I20240503 08:14:24.263520 14557 leader_election.cc:302] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:24.263837 16177 raft_consensus.cc:2781] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:24.263965 16177 raft_consensus.cc:482] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:24.264063 16177 raft_consensus.cc:3037] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.266091 16177 raft_consensus.cc:504] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.266700 16043 raft_consensus.cc:482] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:24.266810 16177 leader_election.cc:288] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:24.266894 16043 raft_consensus.cc:504] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.267289 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a5221747794a13aa4ed763006ba291" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:24.267467 14749 raft_consensus.cc:3037] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.267647 16043 leader_election.cc:288] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:24.268354 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f165b4a5647a48ad7ff951fb3fbf6" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:24.268573 14620 raft_consensus.cc:2454] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:24.268535 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f165b4a5647a48ad7ff951fb3fbf6" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:24.268855 14879 raft_consensus.cc:2454] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:24.269174 14423 leader_election.cc:302] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:24.269572 16043 raft_consensus.cc:2781] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:24.269702 16043 raft_consensus.cc:482] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:24.269824 16043 raft_consensus.cc:3037] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.270001 14749 raft_consensus.cc:2454] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:24.270438 14557 leader_election.cc:302] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:24.270712 16177 raft_consensus.cc:2781] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:24.270869 16177 raft_consensus.cc:686] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:24.271176 16177 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:14:24.273087 16043 raft_consensus.cc:504] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.273906 16043 leader_election.cc:288] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:24.274613 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f165b4a5647a48ad7ff951fb3fbf6" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:24.274814 14620 raft_consensus.cc:3037] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.276468 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da1f165b4a5647a48ad7ff951fb3fbf6" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:24.276757 14879 raft_consensus.cc:3037] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.276883 14344 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:14:24.277437 14620 raft_consensus.cc:2454] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:24.281323 14879 raft_consensus.cc:2454] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:24.275408 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a5221747794a13aa4ed763006ba291" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:24.281718 14489 raft_consensus.cc:3037] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.278141 14423 leader_election.cc:302] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:24.282531 16149 raft_consensus.cc:2781] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:24.282696 16149 raft_consensus.cc:686] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:24.282980 16149 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.285252 14489 raft_consensus.cc:2454] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 1.
I20240503 08:14:24.288085 14344 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:24.351073 16043 raft_consensus.cc:482] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:24.351294 16043 raft_consensus.cc:504] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.352273 16011 raft_consensus.cc:482] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:24.352434 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:24.352427 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:24.352479 16011 raft_consensus.cc:504] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.352643 14879 raft_consensus.cc:2454] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:24.352727 14749 raft_consensus.cc:2454] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:24.353107 14424 leader_election.cc:302] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:24.353150 16011 leader_election.cc:288] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:24.353312 16043 leader_election.cc:288] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:24.353550 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:24.353758 14879 raft_consensus.cc:2454] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:24.353766 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:24.353994 14489 raft_consensus.cc:2454] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:24.354228 16149 raft_consensus.cc:2781] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:24.354202 14685 leader_election.cc:302] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:24.354344 16149 raft_consensus.cc:482] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:24.354426 16149 raft_consensus.cc:3037] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.354565 16011 raft_consensus.cc:2781] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:24.354663 16011 raft_consensus.cc:482] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:24.354769 16011 raft_consensus.cc:3037] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.356504 16011 raft_consensus.cc:504] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.357221 16149 raft_consensus.cc:504] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.357362 16011 leader_election.cc:288] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Requested vote from peers b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:24.357467 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:24.357796 14489 raft_consensus.cc:2379] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in current term 1: Already voted for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in this term.
I20240503 08:14:24.357724 14879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:24.357857 16149 leader_election.cc:288] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:24.357913 14879 raft_consensus.cc:3037] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:24.358793 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:24.358876 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c130eb7ea8a74f4b8d9b79e23af70067" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:24.359038 14749 raft_consensus.cc:2379] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 129a2ac87d7f416db6d47d79bb1c2b5f in current term 1: Already voted for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in this term.
I20240503 08:14:24.359454 14426 leader_election.cc:302] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:14:24.359745 16149 raft_consensus.cc:2726] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:14:24.360455 14879 raft_consensus.cc:2454] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:24.360802 14685 leader_election.cc:302] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:14:24.361073 16011 raft_consensus.cc:2781] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:24.361507 16011 raft_consensus.cc:686] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:24.362108 16011 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:14:24.364996 14344 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:14:24.670207 16177 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:24.676957 16177 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:24.741075 16182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:24.741392 16182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:14:24.743032 16149 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:24.749958 16149 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:24.751503 16182 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:14:24.828004 16180 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:24.857910 16180 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 347.781 ms
  time per row: 0.173891 ms
I20240503 08:14:25.251161 14340 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51430:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:14:25.260270 14730 tablet_service.cc:1467] Processing CreateTablet for tablet b5ceefc466f642d08ea96d549a534776 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.260423 14861 tablet_service.cc:1467] Processing CreateTablet for tablet b5ceefc466f642d08ea96d549a534776 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.260795 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceefc466f642d08ea96d549a534776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.260887 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceefc466f642d08ea96d549a534776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.261755 14601 tablet_service.cc:1467] Processing CreateTablet for tablet dfa711e8c8564bc78846b16d68470cea (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.262212 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa711e8c8564bc78846b16d68470cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.262560 14599 tablet_service.cc:1467] Processing CreateTablet for tablet 88404b9cdf574cb0bca1bb49348fb4cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.262955 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88404b9cdf574cb0bca1bb49348fb4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.266618 14468 tablet_service.cc:1467] Processing CreateTablet for tablet dfa711e8c8564bc78846b16d68470cea (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.267030 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa711e8c8564bc78846b16d68470cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.266953 14600 tablet_service.cc:1467] Processing CreateTablet for tablet b5ceefc466f642d08ea96d549a534776 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.267009 14729 tablet_service.cc:1467] Processing CreateTablet for tablet 88404b9cdf574cb0bca1bb49348fb4cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.267155 14731 tablet_service.cc:1467] Processing CreateTablet for tablet dfa711e8c8564bc78846b16d68470cea (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.267318 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceefc466f642d08ea96d549a534776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.267390 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88404b9cdf574cb0bca1bb49348fb4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.267627 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa711e8c8564bc78846b16d68470cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.274331 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 88404b9cdf574cb0bca1bb49348fb4cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.274844 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88404b9cdf574cb0bca1bb49348fb4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.280416 16250 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:25.282720 16249 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:14:25.284121 16252 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:25.285387 16250 tablet_bootstrap.cc:654] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.285431 16251 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:25.286284 16249 tablet_bootstrap.cc:654] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.286942 16252 tablet_bootstrap.cc:654] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.290704 16251 tablet_bootstrap.cc:654] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.293700 16252 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:25.293853 16252 ts_tablet_manager.cc:1367] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.004s
I20240503 08:14:25.294224 16250 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:25.294399 16249 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: No bootstrap required, opened a new log
I20240503 08:14:25.294477 16250 ts_tablet_manager.cc:1367] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.004s
I20240503 08:14:25.294620 16249 ts_tablet_manager.cc:1367] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240503 08:14:25.294819 16252 raft_consensus.cc:348] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.295229 16252 raft_consensus.cc:374] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.295394 16252 raft_consensus.cc:729] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.295351 16250 raft_consensus.cc:348] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.295385 16249 raft_consensus.cc:348] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.295564 16249 raft_consensus.cc:374] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.295678 16249 raft_consensus.cc:729] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.295781 16250 raft_consensus.cc:374] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.295938 16250 raft_consensus.cc:729] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.295965 16249 consensus_queue.cc:260] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.296066 16252 consensus_queue.cc:260] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.296265 16250 consensus_queue.cc:260] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.296882 16250 ts_tablet_manager.cc:1398] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:14:25.296911 16249 ts_tablet_manager.cc:1398] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:14:25.297235 16250 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:14:25.297782 16251 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:25.297947 16251 ts_tablet_manager.cc:1367] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.000s
I20240503 08:14:25.299095 16250 tablet_bootstrap.cc:654] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.299350 16251 raft_consensus.cc:348] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.299567 16251 raft_consensus.cc:374] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.299669 16251 raft_consensus.cc:729] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.299913 16251 consensus_queue.cc:260] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.300534 16251 ts_tablet_manager.cc:1398] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 08:14:25.300823 16251 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:25.301322 16252 ts_tablet_manager.cc:1398] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240503 08:14:25.301784 16252 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:25.304044 16251 tablet_bootstrap.cc:654] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.306236 16043 raft_consensus.cc:482] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.306404 16043 raft_consensus.cc:504] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.307125 16043 leader_election.cc:288] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:25.307202 16252 tablet_bootstrap.cc:654] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.307857 16250 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: No bootstrap required, opened a new log
I20240503 08:14:25.307922 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa711e8c8564bc78846b16d68470cea" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:25.308180 16250 ts_tablet_manager.cc:1367] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240503 08:14:25.308157 14619 raft_consensus.cc:2454] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 0.
I20240503 08:14:25.308228 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa711e8c8564bc78846b16d68470cea" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:25.308708 14423 leader_election.cc:302] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
W20240503 08:14:25.308885 14426 leader_election.cc:341] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:25.309049 16043 raft_consensus.cc:2781] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:25.309145 16043 raft_consensus.cc:482] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:25.309227 16043 raft_consensus.cc:3037] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.311159 16250 raft_consensus.cc:348] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.311532 16250 raft_consensus.cc:374] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.311643 16250 raft_consensus.cc:729] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.311977 16043 raft_consensus.cc:504] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.311972 16250 consensus_queue.cc:260] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.312883 16250 ts_tablet_manager.cc:1398] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:14:25.313119 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa711e8c8564bc78846b16d68470cea" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:25.313331 14619 raft_consensus.cc:3037] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.313659 16043 leader_election.cc:288] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:25.314677 16252 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:25.314987 16252 ts_tablet_manager.cc:1367] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.008s
I20240503 08:14:25.316147 14619 raft_consensus.cc:2454] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 1.
I20240503 08:14:25.316257 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa711e8c8564bc78846b16d68470cea" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:25.316216 16252 raft_consensus.cc:348] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.316514 16252 raft_consensus.cc:374] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.316599 16252 raft_consensus.cc:729] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.316603 14423 leader_election.cc:302] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:14:25.317035 16043 raft_consensus.cc:2781] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:25.317175 16043 raft_consensus.cc:686] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:25.316987 16252 consensus_queue.cc:260] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
W20240503 08:14:25.317579 14426 leader_election.cc:341] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:25.317739 16252 ts_tablet_manager.cc:1398] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 08:14:25.318034 16252 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:14:25.317461 16043 consensus_queue.cc:237] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.320386 16252 tablet_bootstrap.cc:654] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.323974 16252 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: No bootstrap required, opened a new log
I20240503 08:14:25.323977 16038 raft_consensus.cc:482] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.324182 16251 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:25.324203 16252 ts_tablet_manager.cc:1367] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.003s
I20240503 08:14:25.324162 16038 raft_consensus.cc:504] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.324385 16251 ts_tablet_manager.cc:1367] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.024s	user 0.000s	sys 0.012s
I20240503 08:14:25.324527 14342 catalog_manager.cc:5484] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:14:25.324976 16038 leader_election.cc:288] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:25.324995 16252 raft_consensus.cc:348] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.325106 16251 raft_consensus.cc:348] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.325332 16251 raft_consensus.cc:374] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.325409 16251 raft_consensus.cc:729] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.325600 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88404b9cdf574cb0bca1bb49348fb4cd" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:25.325650 16251 consensus_queue.cc:260] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.325903 14619 raft_consensus.cc:2454] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:25.326368 14814 leader_election.cc:302] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:14:25.326685 16038 raft_consensus.cc:2781] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:25.326788 16038 raft_consensus.cc:482] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:25.326891 16038 raft_consensus.cc:3037] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.329090 16038 raft_consensus.cc:504] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.329633 16038 leader_election.cc:288] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:25.330052 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88404b9cdf574cb0bca1bb49348fb4cd" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:25.325212 16252 raft_consensus.cc:374] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.330346 14619 raft_consensus.cc:3037] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.325717 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88404b9cdf574cb0bca1bb49348fb4cd" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:14:25.330422 16252 raft_consensus.cc:729] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.330263 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88404b9cdf574cb0bca1bb49348fb4cd" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:25.330551 14751 raft_consensus.cc:2454] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 0.
I20240503 08:14:25.330677 16252 consensus_queue.cc:260] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.331265 16252 ts_tablet_manager.cc:1398] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.001s
I20240503 08:14:25.331669 16251 ts_tablet_manager.cc:1398] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240503 08:14:25.331933 16251 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:14:25.332432 14619 raft_consensus.cc:2454] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 1.
I20240503 08:14:25.332832 14814 leader_election.cc:302] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938; no voters: c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:14:25.333146 16038 raft_consensus.cc:2781] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:25.333355 16038 raft_consensus.cc:686] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:14:25.333422 16251 tablet_bootstrap.cc:654] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.333626 16038 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.335976 16251 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: No bootstrap required, opened a new log
I20240503 08:14:25.336237 16251 ts_tablet_manager.cc:1367] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 08:14:25.337553 16251 raft_consensus.cc:348] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.337842 16251 raft_consensus.cc:374] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.337961 16251 raft_consensus.cc:729] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.338287 16251 consensus_queue.cc:260] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.339108 16251 ts_tablet_manager.cc:1398] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:14:25.344419 14344 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:25.356982 16011 raft_consensus.cc:482] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.357196 16011 raft_consensus.cc:504] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.357990 16011 leader_election.cc:288] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:25.358623 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceefc466f642d08ea96d549a534776" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" is_pre_election: true
I20240503 08:14:25.358636 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceefc466f642d08ea96d549a534776" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:25.358866 14619 raft_consensus.cc:2454] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:25.358867 14881 raft_consensus.cc:2454] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 0.
I20240503 08:14:25.359313 14684 leader_election.cc:302] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:25.359627 16011 raft_consensus.cc:2781] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:25.359745 16011 raft_consensus.cc:482] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:25.359815 16011 raft_consensus.cc:3037] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.362681 16011 raft_consensus.cc:504] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.363765 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceefc466f642d08ea96d549a534776" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:25.363986 14620 raft_consensus.cc:3037] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.364156 16011 leader_election.cc:288] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:25.364281 14881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceefc466f642d08ea96d549a534776" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:25.364501 14881 raft_consensus.cc:3037] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.366906 14620 raft_consensus.cc:2454] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:25.367277 14881 raft_consensus.cc:2454] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 1.
I20240503 08:14:25.367772 14685 leader_election.cc:302] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:25.368299 16011 raft_consensus.cc:2781] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:25.368595 16011 raft_consensus.cc:686] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:25.369197 16011 consensus_queue.cc:237] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:14:25.372598 14344 catalog_manager.cc:5484] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:14:25.597469 16011 raft_consensus.cc:482] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.597764 16011 raft_consensus.cc:504] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:14:25.598910 16011 leader_election.cc:288] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:25.599416 14488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa711e8c8564bc78846b16d68470cea" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:14:25.599490 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa711e8c8564bc78846b16d68470cea" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
I20240503 08:14:25.599923 14620 raft_consensus.cc:2379] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in current term 1: Already voted for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in this term.
I20240503 08:14:25.600441 14684 leader_election.cc:302] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:14:25.600730 16011 raft_consensus.cc:3037] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.603222 16011 raft_consensus.cc:2726] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 08:14:25.758548 16265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:25.758924 16265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:25.766525 16265 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:25.785334 16180 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.792836 16011 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 08:14:25.806860 16038 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.813246 14749 raft_consensus.cc:3037] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.816805 16038 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.911005 16149 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.923169 16043 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 371.381 ms
  time per row: 0.18569 ms
I20240503 08:14:26.302976  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:38023
--local_ip_for_outbound_sockets=127.2.49.133
--tserver_master_addrs=127.2.49.190:42751
--webserver_port=36529
--webserver_interface=127.2.49.133
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:14:26.672315 16322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:26.672634 16322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:26.672761 16322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:26.681265 16322 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:26.681491 16322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:26.681697 16322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:14:26.691710 16322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:38023
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=36529
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:26.693037 16322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:26.694821 16322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:26.708567 16328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:26.709316 16329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:26.709856 16331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:26.710582 16322 server_base.cc:1034] running on GCE node
I20240503 08:14:26.711254 16322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:26.712294 16322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:26.713536 16322 hybrid_clock.cc:648] HybridClock initialized: now 1714724066713455 us; error 53 us; skew 500 ppm
I20240503 08:14:26.713882 16322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:26.716367 16322 webserver.cc:460] Webserver started at http://127.2.49.133:36529/ using document root <none> and password file <none>
I20240503 08:14:26.717089 16322 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:26.717222 16322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:26.722767 16322 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240503 08:14:26.725972 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:26.727377 16322 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:14:26.727559 16322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
format_stamp: "Formatted at 2024-05-03 08:14:08 on dist-test-slave-dz87"
I20240503 08:14:26.728380 16322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:26.759979 16322 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:26.760844 16322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:26.761135 16322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:26.762104 16322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:26.763624 16322 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:26.763696 16322 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:26.763818 16322 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:26.763868 16322 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:26.795809 16322 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:38023
I20240503 08:14:26.795887 16449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:38023 every 8 connection(s)
I20240503 08:14:26.797222 16322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:14:26.807404  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 16322
I20240503 08:14:26.808254  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.134:36937
--local_ip_for_outbound_sockets=127.2.49.134
--tserver_master_addrs=127.2.49.190:42751
--webserver_port=35299
--webserver_interface=127.2.49.134
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:26.810546 16450 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:26.810942 16450 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:26.811933 16450 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:26.813627 14341 ts_manager.cc:194] Re-registered known tserver with Master: 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:14:26.814529 14341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:52029
W20240503 08:14:27.174314 16454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:27.174665 16454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:27.174818 16454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:27.182520 16454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:27.182664 16454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:27.182821 16454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.134
I20240503 08:14:27.192195 16454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.134:36937
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.49.134
--webserver_port=35299
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.134
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:27.193526 16454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:27.195348 16454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:27.209565 16460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:27.210106 16463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:27.210673 16461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:27.211408 16454 server_base.cc:1034] running on GCE node
I20240503 08:14:27.212064 16454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:27.213102 16454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:27.214272 16454 hybrid_clock.cc:648] HybridClock initialized: now 1714724067214215 us; error 32 us; skew 500 ppm
I20240503 08:14:27.214602 16454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:27.216925 16454 webserver.cc:460] Webserver started at http://127.2.49.134:35299/ using document root <none> and password file <none>
I20240503 08:14:27.217643 16454 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:27.217779 16454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:27.223196 16454 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:14:27.225837 16469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:27.227178 16454 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:14:27.227352 16454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "95860e22c9474e2d894f04deba8442cc"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:27.228122 16454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:27.253413 16454 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:27.254297 16454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:27.254566 16454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:27.255482 16454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:27.256978 16454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:27.257054 16454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:27.257175 16454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:27.257229 16454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:27.293357 16454 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.134:36937
I20240503 08:14:27.293438 16581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.134:36937 every 8 connection(s)
I20240503 08:14:27.295056 16454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:14:27.304898  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 16454
I20240503 08:14:27.309001 16582 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:27.309361 16582 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:27.310258 16582 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:27.311841 14341 ts_manager.cc:194] Re-registered known tserver with Master: 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937)
I20240503 08:14:27.313148 14341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.134:49493
I20240503 08:14:27.359917  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.135:46299
--local_ip_for_outbound_sockets=127.2.49.135
--tserver_master_addrs=127.2.49.190:42751
--webserver_port=36317
--webserver_interface=127.2.49.135
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:14:27.722759 16586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:27.723137 16586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:27.723286 16586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:27.731014 16586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:27.731144 16586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:27.731292 16586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.135
I20240503 08:14:27.741647 16586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.135:46299
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.49.135
--webserver_port=36317
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.135
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:27.743083 16586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:27.744824 16586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:27.757547 16592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:27.758061 16595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:27.758816 16593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:27.759994 16586 server_base.cc:1034] running on GCE node
I20240503 08:14:27.760658 16586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:27.761662 16586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:27.762859 16586 hybrid_clock.cc:648] HybridClock initialized: now 1714724067762802 us; error 43 us; skew 500 ppm
I20240503 08:14:27.763200 16586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:27.765455 16586 webserver.cc:460] Webserver started at http://127.2.49.135:36317/ using document root <none> and password file <none>
I20240503 08:14:27.766201 16586 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:27.766326 16586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:27.771670 16586 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.004s
I20240503 08:14:27.774493 16601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:27.775740 16586 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240503 08:14:27.775897 16586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6835ad958e094ce4927135b942a2e8a8"
format_stamp: "Formatted at 2024-05-03 08:14:09 on dist-test-slave-dz87"
I20240503 08:14:27.776683 16586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:27.791585 16586 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:27.792423 16586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:27.792704 16586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:27.793632 16586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:27.795138 16586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:27.795209 16586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:27.795320 16586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:27.795385 16586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:27.817756 16450 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
I20240503 08:14:27.828603 16586 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.135:46299
I20240503 08:14:27.828636 16713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.135:46299 every 8 connection(s)
I20240503 08:14:27.830215 16586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:14:27.840226  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 16586
I20240503 08:14:27.841030  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.136:37641
--local_ip_for_outbound_sockets=127.2.49.136
--tserver_master_addrs=127.2.49.190:42751
--webserver_port=40021
--webserver_interface=127.2.49.136
--builtin_ntp_servers=127.2.49.148:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:14:27.846096 16714 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:27.846472 16714 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:27.847404 16714 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:27.848889 14341 ts_manager.cc:194] Re-registered known tserver with Master: 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:14:27.849707 14341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.135:41827
W20240503 08:14:28.206683 16718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:28.206982 16718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:28.207080 16718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:28.215647 16718 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:14:28.215785 16718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:28.215910 16718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.136
I20240503 08:14:28.226038 16718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.136:37641
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.49.136
--webserver_port=40021
--tserver_master_addrs=127.2.49.190:42751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.136
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:28.227402 16718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:28.229063 16718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:28.241922 16724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:28.242624 16727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:28.243181 16725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:28.243850 16718 server_base.cc:1034] running on GCE node
I20240503 08:14:28.244473 16718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:28.245385 16718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:28.246582 16718 hybrid_clock.cc:648] HybridClock initialized: now 1714724068246528 us; error 42 us; skew 500 ppm
I20240503 08:14:28.246887 16718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:28.249044 16718 webserver.cc:460] Webserver started at http://127.2.49.136:40021/ using document root <none> and password file <none>
I20240503 08:14:28.249747 16718 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:28.249871 16718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:28.255002 16718 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:14:28.258009 16733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:28.259461 16718 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:14:28.259615 16718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "036223b0890d4634acfadf9596f39f80"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-dz87"
I20240503 08:14:28.260632 16718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:28.299968 16718 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:14:28.300774 16718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:28.301051 16718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:28.302172 16718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:28.304154 16718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:28.304253 16718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:28.304394 16718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:28.304462 16718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:14:28.316723 16582 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
I20240503 08:14:28.345139 16718 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.136:37641
I20240503 08:14:28.345299 16845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.136:37641 every 8 connection(s)
I20240503 08:14:28.346695 16718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:14:28.348109  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 16718
I20240503 08:14:28.376173 16846 heartbeater.cc:347] Connected to a master server at 127.2.49.190:42751
I20240503 08:14:28.376521 16846 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:28.377477 16846 heartbeater.cc:511] Master 127.2.49.190:42751 requested a full tablet report, sending...
I20240503 08:14:28.379070 14341 ts_manager.cc:194] Re-registered known tserver with Master: 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:14:28.380147 14341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.136:60849
I20240503 08:14:28.852789 16714 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
I20240503 08:14:29.383728 16846 heartbeater.cc:503] Master 127.2.49.190:42751 was elected leader, sending a full tablet report...
I20240503 08:14:29.689693 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.743809 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:29.757577 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:29.802037 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:29.814846 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.826750 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.833663 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.860160 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.866729 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.906216 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.919515 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:29.966508 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:29.986704 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:29.987259 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:30.032927 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:30.105613 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:30.606827 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.632074 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:30.667217 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.692250 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.758407 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.782037 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:30.782640 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:30.789000 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.818950 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:30.821619 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.846226 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.846247 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:30.847048 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:30.895491 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:30.977083 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:30.988972 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:31.176231 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.190519 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.192974 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:31.221055 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:31.233985 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.254612 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:31.289937 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:31.320066 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.506786 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:31.534570 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.557190 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.573832 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.582119 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:31.606256 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:31.611763 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:31.632448 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.901790 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.923249 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:31.932935 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:31.939441 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.968776 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:31.983676 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:31.986646 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.038028 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:32.321993 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.380020 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.392661 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:32.410408 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:32.448403 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.459869 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.481674 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.528357 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:32.551055 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.586854 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:32.650652 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:32.669426 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:32.702466 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.707198 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:32.720737 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:32.729600 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:14:32.734846 14557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.129:37829, user_credentials={real_user=slave}} blocked reactor thread for 43698us
W20240503 08:14:32.751834 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 102887us
I20240503 08:14:33.118525 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:33.126855 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.174726 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:33.181748 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:33.203603 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.206537 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.224596 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.234318 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:33.239845 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:33.247658 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.286734 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.293926 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:33.355024 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.389757 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:33.393239 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:33.397022 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.729272 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:33.749327 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.784425 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.821480 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:33.894287 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:33.912150 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:33.939105 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.081820 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:34.441129 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.455726 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:34.483968 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.511287 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.512784 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:34.544786 14488 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:34.547524 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.585024 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:34.612808 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.615015 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.651510 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:34.652483 14619 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:34.653378 16323 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:34.673283 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.674398 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:34.677017 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.679891 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:34.681273 14751 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:34.702448 16323 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:34.714720 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.735997 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:34.755893 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:14:34.775962 14424 consensus_peers.cc:485] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f65ab6dba1124573962e2da58aa1859c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:34.786564 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.801813 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.805562 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:34.806241 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:34.822469 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:34.831686 17623 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:34.837908 14619 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:34.844357 14345 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:34.856287 14751 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:34.907617 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:34.911867 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:34.961369 14751 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:35.065686 14877 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.066442 17607 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.070911 14488 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.072460 17607 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.112682 17665 ts_tablet_manager.cc:902] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
W20240503 08:14:35.126370 14686 consensus_peers.cc:485] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897a701b880641b08816f2d1a254ef03. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.130600 17669 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.143931 14490 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.145500 17665 tablet_copy_client.cc:308] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:35.156193 14340 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
W20240503 08:14:35.210492 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 67493.7us
I20240503 08:14:35.210572 14510 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet f65ab6dba1124573962e2da58aa1859c from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:49351)
I20240503 08:14:35.210875 14510 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet f65ab6dba1124573962e2da58aa1859c from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:49351: session id = 0f8702fad8ea4d22840dab11bb27f7f2-f65ab6dba1124573962e2da58aa1859c
I20240503 08:14:35.214723 14510 tablet_copy_source_session.cc:205] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:35.215622 14877 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.220925 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65ab6dba1124573962e2da58aa1859c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.236366 17665 tablet_copy_client.cc:783] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:35.244801 17665 tablet_copy_client.cc:647] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:35.255147 17665 tablet_copy_client.cc:523] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:35.263929 17665 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:35.287458 17665 log.cc:824] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:35.289830 14877 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:35.299096 14751 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.302101 16590 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.325459 14619 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.326370 16590 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:35.367913 14816 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55067e08fb694e90b6366d41cc962095. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.369298 14488 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
W20240503 08:14:35.389144 14424 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72a949e76ab5477680a4f9912e9672e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.390293 14751 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.391187 14877 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.391870 16323 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.394374 16323 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.401198 16590 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.412549 14619 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.436985 17730 ts_tablet_manager.cc:902] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:35.440011 17665 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:35.440788 17665 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:35.441426 17665 ts_tablet_manager.cc:1367] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.178s	user 0.078s	sys 0.009s
I20240503 08:14:35.451673 17665 raft_consensus.cc:348] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:35.452368 17665 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.453339 17665 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:35.461720 17665 ts_tablet_manager.cc:1398] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.020s	user 0.016s	sys 0.003s
I20240503 08:14:35.464066 14510 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-f65ab6dba1124573962e2da58aa1859c received from {username='slave'} at 127.2.49.133:49351
I20240503 08:14:35.464272 14510 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-f65ab6dba1124573962e2da58aa1859c on tablet f65ab6dba1124573962e2da58aa1859c with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:35.464920 14348 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.469254 16405 raft_consensus.cc:1203] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:35.470952 14751 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.483358 16323 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:35.499539 14877 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:35.535645 14751 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:35.546435 14340 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:35.610237 17730 tablet_copy_client.cc:308] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:35.618570 14877 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:35.635411 14619 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.636242 16590 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.641206 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 897a701b880641b08816f2d1a254ef03 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:56769)
I20240503 08:14:35.641539 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 897a701b880641b08816f2d1a254ef03 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:56769: session id = 036223b0890d4634acfadf9596f39f80-897a701b880641b08816f2d1a254ef03
I20240503 08:14:35.647706 14751 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.648512 16590 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.651679 14771 tablet_copy_source_session.cc:205] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:35.653446 16590 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.657860 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897a701b880641b08816f2d1a254ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.662317 14751 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.665881 14619 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.670413 14340 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:35.703827 17780 ts_tablet_manager.cc:902] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:35.704396 14488 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:14:35.722328 14816 consensus_peers.cc:485] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88404b9cdf574cb0bca1bb49348fb4cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.732172 14877 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.732941 16323 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.745541 14619 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.746562 16323 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.758548 16323 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.760782 17730 tablet_copy_client.cc:783] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:35.761596 17730 tablet_copy_client.cc:647] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:35.762013 14620 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.765162 14877 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.773131 14488 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:35.773324 14345 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.781659 17780 tablet_copy_client.cc:308] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:35.784539 17730 tablet_copy_client.cc:523] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:35.789196 14878 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.789861 14749 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.792610 17623 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.793426 17623 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.794961 17730 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
W20240503 08:14:35.809183 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30b13cdf4b234f17872f5d1588915968. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:14:35.809604 14425 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc9575f64eb741948e51ccf06f48228a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.810918 17623 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.813536 14878 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.822046 14345 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.823549 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 55067e08fb694e90b6366d41cc962095 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:45301)
I20240503 08:14:35.823850 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 55067e08fb694e90b6366d41cc962095 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:45301: session id = 6835ad958e094ce4927135b942a2e8a8-55067e08fb694e90b6366d41cc962095
I20240503 08:14:35.825341 17730 log.cc:824] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:35.834041 14901 tablet_copy_source_session.cc:205] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:35.841440 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.871826 17780 tablet_copy_client.cc:783] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:35.872480 17780 tablet_copy_client.cc:647] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:35.876755 14749 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:35.879595 17780 tablet_copy_client.cc:523] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:35.884955 17780 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:35.890064 14490 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:35.908466 17780 log.cc:824] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:35.934023 14619 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.934845 16323 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:35.956713 14877 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:35.957418 16323 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:35.964924 14426 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fea213e5a12452f8b894b9cdb94f9ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:35.981124 17623 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:35.984715 14620 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:35.998240 14877 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.000852 14345 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.033222 17845 ts_tablet_manager.cc:902] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:36.049165 17780 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:36.049823 17780 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:36.053079 17780 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.168s	user 0.083s	sys 0.010s
I20240503 08:14:36.072655 17780 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:36.080412 17780 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.082898 17780 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:36.110888 17872 ts_tablet_manager.cc:902] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:36.122589 14877 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:36.126760 17872 tablet_copy_client.cc:308] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:36.127503 16669 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:36.129256 17728 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:36.136523 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55067e08fb694e90b6366d41cc962095. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:36.137431 17780 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.084s	user 0.028s	sys 0.024s
I20240503 08:14:36.137661 14619 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:36.138307 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-55067e08fb694e90b6366d41cc962095 received from {username='slave'} at 127.2.49.135:45301
I20240503 08:14:36.138480 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-55067e08fb694e90b6366d41cc962095 on tablet 55067e08fb694e90b6366d41cc962095 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:36.139497 16590 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:36.142220 14749 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:36.142720 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 88404b9cdf574cb0bca1bb49348fb4cd from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:41491)
I20240503 08:14:36.142913 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 88404b9cdf574cb0bca1bb49348fb4cd from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:41491: session id = 036223b0890d4634acfadf9596f39f80-88404b9cdf574cb0bca1bb49348fb4cd
I20240503 08:14:36.146605 17728 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:36.149427 17728 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.155773 14619 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.158808 14749 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.164165 14901 tablet_copy_source_session.cc:205] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:36.164002 14340 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.166141 17872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88404b9cdf574cb0bca1bb49348fb4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.171972 17845 tablet_copy_client.cc:308] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:36.172720 14510 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 72a949e76ab5477680a4f9912e9672e9 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:49351)
I20240503 08:14:36.172972 14510 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 72a949e76ab5477680a4f9912e9672e9 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:49351: session id = 0f8702fad8ea4d22840dab11bb27f7f2-72a949e76ab5477680a4f9912e9672e9
I20240503 08:14:36.176589 17872 tablet_copy_client.cc:783] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:36.176952 17872 tablet_copy_client.cc:647] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.177111 14510 tablet_copy_source_session.cc:205] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:36.178529 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.179373 17872 tablet_copy_client.cc:523] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.181774 17780 ts_tablet_manager.cc:902] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:36.184590 17872 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:36.193265 17845 tablet_copy_client.cc:783] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:36.193572 17845 tablet_copy_client.cc:647] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.199330 16668 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.200476 17845 tablet_copy_client.cc:523] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.204398 17845 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:36.212615 17898 ts_tablet_manager.cc:902] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:36.235968 17730 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:36.238855 17730 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:36.239527 17730 ts_tablet_manager.cc:1367] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.445s	user 0.063s	sys 0.020s
I20240503 08:14:36.240805 14749 consensus_queue.cc:237] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:36.253491 17730 raft_consensus.cc:348] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:36.259039 14619 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:36.259802 17730 raft_consensus.cc:729] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.259819 17674 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:36.261695 17730 consensus_queue.cc:260] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:36.268514 14877 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:36.269196 17674 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:36.273303 17730 ts_tablet_manager.cc:1398] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.007s
I20240503 08:14:36.283864 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-897a701b880641b08816f2d1a254ef03 received from {username='slave'} at 127.2.49.136:56769
I20240503 08:14:36.284080 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 036223b0890d4634acfadf9596f39f80-897a701b880641b08816f2d1a254ef03 on tablet 897a701b880641b08816f2d1a254ef03 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:36.287830 17780 tablet_copy_client.cc:308] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:36.296991 17918 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:36.303189 14510 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet dc9575f64eb741948e51ccf06f48228a from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:14:36.303537 14510 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet dc9575f64eb741948e51ccf06f48228a from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-dc9575f64eb741948e51ccf06f48228a
I20240503 08:14:36.313628 14510 tablet_copy_source_session.cc:205] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:36.318177 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9575f64eb741948e51ccf06f48228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.319977 14877 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:36.331015 14345 catalog_manager.cc:5484] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:14:36.342226 14685 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5ceefc466f642d08ea96d549a534776. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:36.364935 17780 tablet_copy_client.cc:783] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:36.371520 17780 tablet_copy_client.cc:647] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.387250 14619 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:36.390998 17780 tablet_copy_client.cc:523] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.394066 17898 tablet_copy_client.cc:308] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:36.394856 14510 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 30b13cdf4b234f17872f5d1588915968 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:14:36.395062 14510 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 30b13cdf4b234f17872f5d1588915968 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-30b13cdf4b234f17872f5d1588915968
I20240503 08:14:36.397980 14510 tablet_copy_source_session.cc:205] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:36.399793 17898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b13cdf4b234f17872f5d1588915968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.402643 17780 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:36.402863 16801 raft_consensus.cc:1203] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:36.407199 17898 tablet_copy_client.cc:783] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:36.407459 17898 tablet_copy_client.cc:647] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.412875 17898 tablet_copy_client.cc:523] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.424499 17898 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:36.542922 17954 ts_tablet_manager.cc:902] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:36.557515 17872 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:36.566457 17845 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:36.569147 17845 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:36.571704 17872 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:36.608165 17954 tablet_copy_client.cc:308] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:36.621455 17845 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.417s	user 0.057s	sys 0.009s
I20240503 08:14:36.626871 17872 ts_tablet_manager.cc:1367] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.442s	user 0.072s	sys 0.024s
I20240503 08:14:36.636788 17872 raft_consensus.cc:348] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:36.649297 17845 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:36.655145 17845 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.663023 17845 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:36.665220 17872 raft_consensus.cc:729] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.665625 17872 consensus_queue.cc:260] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:36.666461 17872 ts_tablet_manager.cc:1398] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.031s	user 0.002s	sys 0.000s
I20240503 08:14:36.667783 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-88404b9cdf574cb0bca1bb49348fb4cd received from {username='slave'} at 127.2.49.136:41491
I20240503 08:14:36.667927 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 036223b0890d4634acfadf9596f39f80-88404b9cdf574cb0bca1bb49348fb4cd on tablet 88404b9cdf574cb0bca1bb49348fb4cd with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:36.671512 17780 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:36.672019 17780 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:36.672343 17780 ts_tablet_manager.cc:1367] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.270s	user 0.059s	sys 0.009s
I20240503 08:14:36.675386 17898 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:36.675788 17898 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:36.676076 17898 ts_tablet_manager.cc:1367] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.255s	user 0.041s	sys 0.021s
I20240503 08:14:36.676687 17780 raft_consensus.cc:348] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:36.676946 17780 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.677209 17780 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:36.678058 17898 raft_consensus.cc:348] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:36.678678 17898 raft_consensus.cc:729] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:36.680498 17780 ts_tablet_manager.cc:1398] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20240503 08:14:36.680898 17898 consensus_queue.cc:260] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:36.681553 17898 ts_tablet_manager.cc:1398] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240503 08:14:36.681586 17845 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.000s
I20240503 08:14:36.688941 14510 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 3fea213e5a12452f8b894b9cdb94f9ae from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:50025)
I20240503 08:14:36.689003 14509 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-72a949e76ab5477680a4f9912e9672e9 received from {username='slave'} at 127.2.49.133:49351
I20240503 08:14:36.689121 14509 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-72a949e76ab5477680a4f9912e9672e9 on tablet 72a949e76ab5477680a4f9912e9672e9 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:36.689440 14510 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 3fea213e5a12452f8b894b9cdb94f9ae from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:50025: session id = 95860e22c9474e2d894f04deba8442cc-3fea213e5a12452f8b894b9cdb94f9ae
I20240503 08:14:36.689755 16669 raft_consensus.cc:1203] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:36.692111 14619 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:36.692550 14510 tablet_copy_source_session.cc:205] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:36.693101 14508 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-dc9575f64eb741948e51ccf06f48228a received from {username='slave'} at 127.2.49.135:44583
I20240503 08:14:36.693209 14508 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-dc9575f64eb741948e51ccf06f48228a on tablet dc9575f64eb741948e51ccf06f48228a with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:36.693744 17990 ts_tablet_manager.cc:902] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:36.694123 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-30b13cdf4b234f17872f5d1588915968 received from {username='slave'} at 127.2.49.135:44583
I20240503 08:14:36.694247 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-30b13cdf4b234f17872f5d1588915968 on tablet 30b13cdf4b234f17872f5d1588915968 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:36.698441 14490 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:36.699446 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea213e5a12452f8b894b9cdb94f9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.723317 17845 ts_tablet_manager.cc:902] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:36.728547 14751 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:36.729878 17516 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:36.737449 14749 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:36.742771 14557 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c595bb19d6946b0b1f853064892e946. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:36.743041 14488 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:36.745141 17623 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
W20240503 08:14:36.750687 14424 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87c5af03f4d47d38bbfbd225c7195d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:36.751577 14877 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:36.753041 17516 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:36.759763 17954 tablet_copy_client.cc:783] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:36.761121 17954 tablet_copy_client.cc:647] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.763526 18010 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:36.765053 17954 tablet_copy_client.cc:523] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.775830 17954 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:36.778256 18000 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.782552 14490 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.791895 14748 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.795560 14345 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:36.806831 16669 raft_consensus.cc:1203] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:36.826864 17954 log.cc:824] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:36.841957 17845 tablet_copy_client.cc:308] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:36.847699 17990 tablet_copy_client.cc:308] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:36.919587 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:36.920830 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:36.922746 16405 raft_consensus.cc:1203] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:36.926093 16323 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:36.939657 14748 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:36.941324 14346 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:36.948917 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet b5ceefc466f642d08ea96d549a534776 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:37499)
I20240503 08:14:36.949127 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet b5ceefc466f642d08ea96d549a534776 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:37499: session id = 0f8702fad8ea4d22840dab11bb27f7f2-b5ceefc466f642d08ea96d549a534776
I20240503 08:14:36.953052 14771 tablet_copy_source_session.cc:205] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:36.956154 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceefc466f642d08ea96d549a534776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:36.966310 17845 tablet_copy_client.cc:783] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:36.968864 17845 tablet_copy_client.cc:647] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:36.973775 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:36.974488 17845 tablet_copy_client.cc:523] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:36.987607 17845 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:36.987699 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.026748 14879 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:37.032269 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:37.047978 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:37.096060 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.118237 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 55067e08fb694e90b6366d41cc962095 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:44455)
I20240503 08:14:37.118439 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 55067e08fb694e90b6366d41cc962095 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:44455: session id = 95860e22c9474e2d894f04deba8442cc-55067e08fb694e90b6366d41cc962095
I20240503 08:14:37.121459 14901 tablet_copy_source_session.cc:205] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:37.126008 17954 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:37.129793 17954 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:37.130499 17954 ts_tablet_manager.cc:1367] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.355s	user 0.075s	sys 0.014s
I20240503 08:14:37.133816 17990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:37.153177 17990 tablet_copy_client.cc:783] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:37.153517 17990 tablet_copy_client.cc:647] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:37.165076 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.167371 17954 raft_consensus.cc:348] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:37.168012 17954 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:37.178833 17954 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:37.184918 17954 ts_tablet_manager.cc:1398] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.054s	user 0.009s	sys 0.007s
I20240503 08:14:37.185990 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-3fea213e5a12452f8b894b9cdb94f9ae received from {username='slave'} at 127.2.49.134:50025
I20240503 08:14:37.186129 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-3fea213e5a12452f8b894b9cdb94f9ae on tablet 3fea213e5a12452f8b894b9cdb94f9ae with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:37.203720 17990 tablet_copy_client.cc:523] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:37.209947 17990 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:37.231247 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.238744 16801 raft_consensus.cc:1203] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 08:14:37.249133 14687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.129:37829, user_credentials={real_user=slave}} blocked reactor thread for 98282.1us
I20240503 08:14:37.306535 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:37.307893 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.308682 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:37.319424 18088 ts_tablet_manager.cc:902] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:37.328071 17954 ts_tablet_manager.cc:902] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:37.403116 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.445670 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:37.460011 17954 tablet_copy_client.cc:308] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:37.466899 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 5c595bb19d6946b0b1f853064892e946 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:48741)
I20240503 08:14:37.467134 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 5c595bb19d6946b0b1f853064892e946 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:48741: session id = 95860e22c9474e2d894f04deba8442cc-5c595bb19d6946b0b1f853064892e946
I20240503 08:14:37.470280 14641 tablet_copy_source_session.cc:205] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:37.472043 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c595bb19d6946b0b1f853064892e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:37.479425 17954 tablet_copy_client.cc:783] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:37.479948 17954 tablet_copy_client.cc:647] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:37.483112 17954 tablet_copy_client.cc:523] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:37.498718 17954 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:37.499364 14489 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:37.500512 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.547542 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:37.593456 16537 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:37.594785 18010 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:37.607540 14619 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:37.608395 18010 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:37.615053 14877 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:37.615839 18010 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:37.639648 16323 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.642624 14748 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:37.644771 16536 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.651491 14346 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
W20240503 08:14:37.657554 14686 consensus_peers.cc:485] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c130eb7ea8a74f4b8d9b79e23af70067. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:37.659010 14879 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:37.659608 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.659673 17927 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:37.660717 14489 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:37.661284 17927 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:37.679029 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:14:37.688098 14425 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fea213e5a12452f8b894b9cdb94f9ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:37.692042 14879 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.693495 17927 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.695832 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:37.700599 14489 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.702509 14348 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.716493 14877 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.738341 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.746541 17845 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:37.746994 17845 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:37.765269 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:37.766845 17845 ts_tablet_manager.cc:1367] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.779s	user 0.098s	sys 0.128s
I20240503 08:14:37.767158 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:37.796507 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:37.811278 17845 raft_consensus.cc:348] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:37.811580 17845 raft_consensus.cc:729] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:37.811908 17845 consensus_queue.cc:260] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:37.823748 14619 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:37.829649 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:37.854588 17845 ts_tablet_manager.cc:1398] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.088s	user 0.001s	sys 0.001s
I20240503 08:14:37.857296 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-b5ceefc466f642d08ea96d549a534776 received from {username='slave'} at 127.2.49.133:37499
I20240503 08:14:37.857452 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-b5ceefc466f642d08ea96d549a534776 on tablet b5ceefc466f642d08ea96d549a534776 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:37.861160 16405 raft_consensus.cc:1203] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:37.861764 17954 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:37.862274 17954 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:37.862534 17954 ts_tablet_manager.cc:1367] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.364s	user 0.061s	sys 0.005s
I20240503 08:14:37.863384 17954 raft_consensus.cc:348] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:37.863588 17954 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:37.863850 17954 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:37.870743 18088 tablet_copy_client.cc:308] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:37.873594 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet a87c5af03f4d47d38bbfbd225c7195d1 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:49351)
I20240503 08:14:37.873787 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet a87c5af03f4d47d38bbfbd225c7195d1 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:49351: session id = 0f8702fad8ea4d22840dab11bb27f7f2-a87c5af03f4d47d38bbfbd225c7195d1
I20240503 08:14:37.876871 14507 tablet_copy_source_session.cc:205] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:37.879268 17954 ts_tablet_manager.cc:1398] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.017s	user 0.000s	sys 0.002s
I20240503 08:14:37.880050 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-5c595bb19d6946b0b1f853064892e946 received from {username='slave'} at 127.2.49.134:48741
I20240503 08:14:37.880251 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-5c595bb19d6946b0b1f853064892e946 on tablet 5c595bb19d6946b0b1f853064892e946 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:37.909926 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c5af03f4d47d38bbfbd225c7195d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:37.943642 16537 raft_consensus.cc:1203] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:37.947739 18088 tablet_copy_client.cc:783] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:37.955515 18088 tablet_copy_client.cc:647] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:37.959482 17990 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:37.959965 17990 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:37.960361 17990 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.751s	user 0.060s	sys 0.000s
I20240503 08:14:37.960647 18088 tablet_copy_client.cc:523] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:37.961375 17990 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:37.961678 17990 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:37.961985 17990 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:37.962931 17990 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:37.963740 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-55067e08fb694e90b6366d41cc962095 received from {username='slave'} at 127.2.49.134:44455
I20240503 08:14:37.963903 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-55067e08fb694e90b6366d41cc962095 on tablet 55067e08fb694e90b6366d41cc962095 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:37.980674 18088 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:38.015846 18169 ts_tablet_manager.cc:902] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:38.036003 18169 tablet_copy_client.cc:308] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:38.080871 18186 ts_tablet_manager.cc:902] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:38.088440 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 3fea213e5a12452f8b894b9cdb94f9ae from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:55303)
I20240503 08:14:38.088644 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 3fea213e5a12452f8b894b9cdb94f9ae from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:55303: session id = 036223b0890d4634acfadf9596f39f80-3fea213e5a12452f8b894b9cdb94f9ae
I20240503 08:14:38.091759 14507 tablet_copy_source_session.cc:205] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:38.093420 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea213e5a12452f8b894b9cdb94f9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:38.125865 18186 tablet_copy_client.cc:308] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:38.127439 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet c130eb7ea8a74f4b8d9b79e23af70067 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:56769)
I20240503 08:14:38.127643 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet c130eb7ea8a74f4b8d9b79e23af70067 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:56769: session id = 036223b0890d4634acfadf9596f39f80-c130eb7ea8a74f4b8d9b79e23af70067
I20240503 08:14:38.132686 14771 tablet_copy_source_session.cc:205] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:38.134886 18186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130eb7ea8a74f4b8d9b79e23af70067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:38.137499 18169 tablet_copy_client.cc:783] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:38.137951 18169 tablet_copy_client.cc:647] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:38.145471 14490 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:14:38.158746 14425 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e86f4f3a01048e69f553c6557d79b71. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.159813 14620 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.160497 18148 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.163353 14748 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.164037 18148 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.171546 18148 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.172158 18186 tablet_copy_client.cc:783] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:38.173655 18186 tablet_copy_client.cc:647] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:38.174592 14751 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.181460 18169 tablet_copy_client.cc:523] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:38.181839 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.188532 14620 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.189002 18169 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:38.199911 18088 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:38.200429 18088 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:38.200711 18088 ts_tablet_manager.cc:1367] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.225s	user 0.064s	sys 0.020s
I20240503 08:14:38.201560 18088 raft_consensus.cc:348] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:38.201835 18088 raft_consensus.cc:729] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:38.202112 18088 consensus_queue.cc:260] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:38.202750 18088 ts_tablet_manager.cc:1398] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:14:38.203400 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-a87c5af03f4d47d38bbfbd225c7195d1 received from {username='slave'} at 127.2.49.133:49351
I20240503 08:14:38.203537 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-a87c5af03f4d47d38bbfbd225c7195d1 on tablet a87c5af03f4d47d38bbfbd225c7195d1 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:38.203567 18186 tablet_copy_client.cc:523] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:38.228283 18186 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:38.240685 14620 consensus_queue.cc:237] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:38.260931 14490 raft_consensus.cc:1261] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.261945 18000 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.266986 16537 raft_consensus.cc:1203] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:38.271047 14877 raft_consensus.cc:1261] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.272274 18000 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.314683 18241 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.333488 14879 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
W20240503 08:14:38.337779 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76a216295a8a4feaa2d3326035879073. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.340340 14345 catalog_manager.cc:5484] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.343750 14490 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:38.375633 14490 consensus_queue.cc:237] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:38.386900 18169 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:38.389393 18169 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:38.397903 18169 ts_tablet_manager.cc:1367] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.209s	user 0.076s	sys 0.008s
I20240503 08:14:38.400894 18169 raft_consensus.cc:348] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
W20240503 08:14:38.403919 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c687520c0a349fd8438dacbd5011001. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.404224 18169 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:38.404943 14751 raft_consensus.cc:1261] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.405537 18148 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.407951 18169 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:38.423974 18169 ts_tablet_manager.cc:1398] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.026s	user 0.004s	sys 0.000s
I20240503 08:14:38.425026 16801 raft_consensus.cc:1203] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:38.432006 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-3fea213e5a12452f8b894b9cdb94f9ae received from {username='slave'} at 127.2.49.136:55303
I20240503 08:14:38.432157 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 036223b0890d4634acfadf9596f39f80-3fea213e5a12452f8b894b9cdb94f9ae on tablet 3fea213e5a12452f8b894b9cdb94f9ae with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:38.453238 14879 raft_consensus.cc:1261] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.458185 18148 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.475935 18148 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.478114 14877 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.484421 14345 catalog_manager.cc:5484] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.487982 14751 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.523764 18186 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:38.525436 18186 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:38.525790 18186 ts_tablet_manager.cc:1367] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.298s	user 0.053s	sys 0.012s
I20240503 08:14:38.526861 18186 raft_consensus.cc:348] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:38.527171 18186 raft_consensus.cc:729] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:38.527514 18186 consensus_queue.cc:260] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:38.528301 18186 ts_tablet_manager.cc:1398] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:38.529187 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-c130eb7ea8a74f4b8d9b79e23af70067 received from {username='slave'} at 127.2.49.136:56769
I20240503 08:14:38.529376 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 036223b0890d4634acfadf9596f39f80-c130eb7ea8a74f4b8d9b79e23af70067 on tablet c130eb7ea8a74f4b8d9b79e23af70067 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:38.558358 14751 consensus_queue.cc:237] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:38.585866 16405 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:38.586052 14877 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:38.586696 17927 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:38.586915 14620 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:38.588557 17927 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:38.591634 17927 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:38.638851 18271 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.639653 14619 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.649749 14345 catalog_manager.cc:5484] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.657196 14879 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:38.659062 16405 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
W20240503 08:14:38.711293 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5ceefc466f642d08ea96d549a534776. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.759292 16405 raft_consensus.cc:1203] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:38.782809 18328 ts_tablet_manager.cc:902] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:38.816293 14620 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:38.816885 18337 ts_tablet_manager.cc:902] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:38.831571 14490 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.833563 18241 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.846084 14751 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.847229 18239 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.860394 18328 tablet_copy_client.cc:308] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:38.861322 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:14:38.861505 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-4e86f4f3a01048e69f553c6557d79b71
W20240503 08:14:38.862205 14555 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd11460051ab40febc2fe6aff58b5028. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.864408 14507 tablet_copy_source_session.cc:205] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:38.866470 18328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:38.868106 18344 ts_tablet_manager.cc:902] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:38.875116 18328 tablet_copy_client.cc:783] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:38.875420 18328 tablet_copy_client.cc:647] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:38.877745 18328 tablet_copy_client.cc:523] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:38.879624 18241 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:38.882618 14751 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:38.884054 18328 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:38.887481 18337 tablet_copy_client.cc:308] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:38.888269 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 1c687520c0a349fd8438dacbd5011001 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:50025)
I20240503 08:14:38.888454 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 1c687520c0a349fd8438dacbd5011001 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:50025: session id = 95860e22c9474e2d894f04deba8442cc-1c687520c0a349fd8438dacbd5011001
I20240503 08:14:38.891431 14507 tablet_copy_source_session.cc:205] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:38.893959 18337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c687520c0a349fd8438dacbd5011001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:38.893925 14345 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:38.907756 18337 tablet_copy_client.cc:783] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:38.908684 18337 tablet_copy_client.cc:647] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:38.908680 14490 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:38.916724 18337 tablet_copy_client.cc:523] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:38.921476 14620 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:38.927781 18337 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:38.944363 14748 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.944823 14879 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:38.945410 18241 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.946902 18239 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:38.951211 18344 tablet_copy_client.cc:308] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:38.968245 18241 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
W20240503 08:14:38.980094 14556 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 169bd9ea6772420abf757dd55cf44d31. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:38.990530 14749 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:38.999066 14879 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:39.007530 14345 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:39.037401 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 76a216295a8a4feaa2d3326035879073 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:57463)
I20240503 08:14:39.037652 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 76a216295a8a4feaa2d3326035879073 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:57463: session id = 6835ad958e094ce4927135b942a2e8a8-76a216295a8a4feaa2d3326035879073
I20240503 08:14:39.042706 14641 tablet_copy_source_session.cc:205] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:39.044690 18344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a216295a8a4feaa2d3326035879073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:39.059660 18344 tablet_copy_client.cc:783] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:39.060292 18344 tablet_copy_client.cc:647] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:39.064348 18344 tablet_copy_client.cc:523] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:39.070766 18344 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:39.074684 16801 raft_consensus.cc:1203] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:39.090610 18389 ts_tablet_manager.cc:902] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:39.104465 18328 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:39.105579 18328 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:39.106858 18328 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.223s	user 0.076s	sys 0.006s
I20240503 08:14:39.111526 18328 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:39.113241 18328 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:39.113860 18328 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:39.118809 18389 tablet_copy_client.cc:308] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:39.130214 18337 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:39.130749 18337 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:39.131124 18337 ts_tablet_manager.cc:1367] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.203s	user 0.050s	sys 0.017s
I20240503 08:14:39.139673 18337 raft_consensus.cc:348] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:39.140076 18337 raft_consensus.cc:729] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:39.140343 18337 consensus_queue.cc:260] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:39.177479 18328 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.069s	user 0.021s	sys 0.025s
I20240503 08:14:39.182838 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-4e86f4f3a01048e69f553c6557d79b71 received from {username='slave'} at 127.2.49.135:44583
I20240503 08:14:39.182993 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-4e86f4f3a01048e69f553c6557d79b71 on tablet 4e86f4f3a01048e69f553c6557d79b71 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:39.185983 18337 ts_tablet_manager.cc:1398] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.055s	user 0.005s	sys 0.004s
I20240503 08:14:39.187855 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-1c687520c0a349fd8438dacbd5011001 received from {username='slave'} at 127.2.49.134:50025
I20240503 08:14:39.187985 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-1c687520c0a349fd8438dacbd5011001 on tablet 1c687520c0a349fd8438dacbd5011001 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:39.205768 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet b5ceefc466f642d08ea96d549a534776 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:44645)
I20240503 08:14:39.206025 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet b5ceefc466f642d08ea96d549a534776 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:44645: session id = 95860e22c9474e2d894f04deba8442cc-b5ceefc466f642d08ea96d549a534776
I20240503 08:14:39.209522 14771 tablet_copy_source_session.cc:205] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:39.212194 18389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceefc466f642d08ea96d549a534776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:39.219683 18389 tablet_copy_client.cc:783] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:39.220127 18389 tablet_copy_client.cc:647] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:39.222009 18389 tablet_copy_client.cc:523] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:39.237229 18389 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:39.258064 16537 raft_consensus.cc:1203] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:39.314066 16668 raft_consensus.cc:1203] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:39.343226 18344 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:39.343860 18344 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:39.344158 18344 ts_tablet_manager.cc:1367] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.274s	user 0.045s	sys 0.011s
I20240503 08:14:39.344935 18344 raft_consensus.cc:348] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:39.345192 18344 raft_consensus.cc:729] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:39.345446 18344 consensus_queue.cc:260] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:39.346242 18344 ts_tablet_manager.cc:1398] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:14:39.346544 18389 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:39.346949 18389 tablet_bootstrap.cc:492] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:39.347223 18389 ts_tablet_manager.cc:1367] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.110s	user 0.052s	sys 0.008s
I20240503 08:14:39.347203 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-76a216295a8a4feaa2d3326035879073 received from {username='slave'} at 127.2.49.135:57463
I20240503 08:14:39.347368 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-76a216295a8a4feaa2d3326035879073 on tablet 76a216295a8a4feaa2d3326035879073 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:39.348022 18389 raft_consensus.cc:348] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:39.348240 18389 raft_consensus.cc:729] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:39.348477 18389 consensus_queue.cc:260] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:39.369786 18389 ts_tablet_manager.cc:1398] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.022s	user 0.002s	sys 0.001s
I20240503 08:14:39.370635 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-b5ceefc466f642d08ea96d549a534776 received from {username='slave'} at 127.2.49.134:44645
I20240503 08:14:39.370762 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-b5ceefc466f642d08ea96d549a534776 on tablet b5ceefc466f642d08ea96d549a534776 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:39.382555 18470 ts_tablet_manager.cc:902] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:39.385200 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:39.389003 18470 tablet_copy_client.cc:308] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:39.404372 16668 raft_consensus.cc:1203] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:39.448670 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.460590 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.464241 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.469367 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:14:39.507099 14425 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da1f165b4a5647a48ad7ff951fb3fbf6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:39.508348 14877 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:39.510243 14620 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:39.510963 18148 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:39.511525 18148 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:39.517146 18491 ts_tablet_manager.cc:902] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:39.526120 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:39.538779 18139 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.542472 14879 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.548889 14348 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.554562 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.563704 14620 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:14:39.596467 14424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 47087.9us
I20240503 08:14:39.602447 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet dd11460051ab40febc2fe6aff58b5028 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:53581)
I20240503 08:14:39.602639 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet dd11460051ab40febc2fe6aff58b5028 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:53581: session id = 0f8702fad8ea4d22840dab11bb27f7f2-dd11460051ab40febc2fe6aff58b5028
I20240503 08:14:39.605535 14641 tablet_copy_source_session.cc:205] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:39.609066 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd11460051ab40febc2fe6aff58b5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:39.614089 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:39.616331 18470 tablet_copy_client.cc:783] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:39.616693 18470 tablet_copy_client.cc:647] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:39.617246 14619 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.622845 18470 tablet_copy_client.cc:523] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:14:39.623680 14556 consensus_peers.cc:485] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7a5221747794a13aa4ed763006ba291. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:39.625001 14748 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:39.625569 18239 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:39.628036 14490 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:39.639849 18239 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:39.646423 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:39.651822 18470 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:39.652067 16537 raft_consensus.cc:1203] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:39.654582 18491 tablet_copy_client.cc:308] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:39.664810 18239 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.685463 14748 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.686839 14490 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.692246 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.702868 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:39.709196 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.710963 14346 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index -1 to 4, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:39.764475 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:39.773667 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.790307 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:39.839429 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:39.849205 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.849753 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.855979 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 169bd9ea6772420abf757dd55cf44d31 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:34545)
I20240503 08:14:39.856168 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 169bd9ea6772420abf757dd55cf44d31 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:34545: session id = 036223b0890d4634acfadf9596f39f80-169bd9ea6772420abf757dd55cf44d31
I20240503 08:14:39.856803 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:39.859494 14641 tablet_copy_source_session.cc:205] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:39.861234 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:39.873565 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.885680 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.887300 18491 tablet_copy_client.cc:783] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:39.887607 18491 tablet_copy_client.cc:647] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:39.907944 18491 tablet_copy_client.cc:523] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:39.917052 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:39.940255 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.949002 18491 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:39.966782 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:39.985651 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:40.066359 18566 ts_tablet_manager.cc:902] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:40.099030 18566 tablet_copy_client.cc:308] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:40.099838 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:14:40.100032 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:14:40.103679 14507 tablet_copy_source_session.cc:205] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.106662 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.139047 18566 tablet_copy_client.cc:783] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.140282 14620 consensus_queue.cc:237] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:40.150480 18566 tablet_copy_client.cc:647] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.169107 18566 tablet_copy_client.cc:523] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.176753 14490 consensus_queue.cc:237] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:40.179867 18566 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
W20240503 08:14:40.183665 14556 consensus_peers.cc:485] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b19df3ae9f2b44a3818855f259962fff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.184091 18582 ts_tablet_manager.cc:902] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:40.185086 14877 raft_consensus.cc:1261] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.185710 18239 consensus_queue.cc:1035] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.192036 14489 raft_consensus.cc:1261] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.193017 18239 consensus_queue.cc:1035] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.194325 14619 raft_consensus.cc:1261] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:40.194751 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa711e8c8564bc78846b16d68470cea. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.201484 18501 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240503 08:14:40.204486 18239 raft_consensus.cc:2932] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.207964 14748 raft_consensus.cc:1261] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.209301 18501 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.211709 14489 raft_consensus.cc:2932] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.217854 18582 tablet_copy_client.cc:308] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:40.220089 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet f7a5221747794a13aa4ed763006ba291 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:57463)
I20240503 08:14:40.220264 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet f7a5221747794a13aa4ed763006ba291 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:57463: session id = 6835ad958e094ce4927135b942a2e8a8-f7a5221747794a13aa4ed763006ba291
I20240503 08:14:40.232455 14641 tablet_copy_source_session.cc:205] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.234642 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a5221747794a13aa4ed763006ba291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.239663 18501 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.241364 18582 tablet_copy_client.cc:783] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.241730 18582 tablet_copy_client.cc:647] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.243965 14620 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.250057 14340 catalog_manager.cc:5484] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.257067 14877 raft_consensus.cc:2932] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.262660 18582 tablet_copy_client.cc:523] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.269563 18582 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:40.269747 18491 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.270251 18491 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:40.270530 18491 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.322s	user 0.047s	sys 0.006s
I20240503 08:14:40.271384 18491 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:40.271636 18491 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.271901 18491 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:40.279922 18491 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20240503 08:14:40.288223 18470 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.288731 18470 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:40.289093 18470 ts_tablet_manager.cc:1367] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.637s	user 0.055s	sys 0.001s
I20240503 08:14:40.289903 18470 raft_consensus.cc:348] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:40.290143 18470 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.290446 18470 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:40.299443 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-169bd9ea6772420abf757dd55cf44d31 received from {username='slave'} at 127.2.49.136:34545
I20240503 08:14:40.299727 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 036223b0890d4634acfadf9596f39f80-169bd9ea6772420abf757dd55cf44d31 on tablet 169bd9ea6772420abf757dd55cf44d31 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:40.315604 14751 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.331763 14342 catalog_manager.cc:5484] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.408298 16800 raft_consensus.cc:1203] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:40.479751 18470 ts_tablet_manager.cc:1398] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.191s	user 0.057s	sys 0.085s
I20240503 08:14:40.483033 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-dd11460051ab40febc2fe6aff58b5028 received from {username='slave'} at 127.2.49.133:53581
I20240503 08:14:40.483211 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-dd11460051ab40febc2fe6aff58b5028 on tablet dd11460051ab40febc2fe6aff58b5028 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:40.604086 14489 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:40.625586 18566 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.626116 18566 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:40.626443 18566 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.447s	user 0.053s	sys 0.007s
I20240503 08:14:40.627275 18566 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:40.627559 18566 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.627808 18566 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:40.628186 18582 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.628600 18582 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:40.628898 18582 ts_tablet_manager.cc:1367] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.359s	user 0.054s	sys 0.003s
I20240503 08:14:40.629794 18582 raft_consensus.cc:348] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:40.630065 18582 raft_consensus.cc:729] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.639925 18566 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.002s
I20240503 08:14:40.640923 14620 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:40.642544 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-da1f165b4a5647a48ad7ff951fb3fbf6 received from {username='slave'} at 127.2.49.135:44583
I20240503 08:14:40.642696 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-da1f165b4a5647a48ad7ff951fb3fbf6 on tablet da1f165b4a5647a48ad7ff951fb3fbf6 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:40.642113 18656 ts_tablet_manager.cc:902] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
W20240503 08:14:40.645097 14425 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b677894b51f4dc58f157d8f4d632e9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.645937 14619 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.646770 18602 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.648998 14751 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.649611 18239 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.652155 14879 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.652801 18505 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.657271 14877 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.659868 18239 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:40.660768 14555 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 562e1ec5c34945298cf457758328057f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.665128 18505 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.669593 14879 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.673595 14619 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.675483 18582 consensus_queue.cc:260] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:40.676141 18582 ts_tablet_manager.cc:1398] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.047s	user 0.004s	sys 0.000s
I20240503 08:14:40.678997 14342 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:40.681988 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-f7a5221747794a13aa4ed763006ba291 received from {username='slave'} at 127.2.49.135:57463
I20240503 08:14:40.682142 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-f7a5221747794a13aa4ed763006ba291 on tablet f7a5221747794a13aa4ed763006ba291 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:40.684259 18241 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:40.697979 14878 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:40.700363 18656 tablet_copy_client.cc:308] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:40.701102 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet dfa711e8c8564bc78846b16d68470cea from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:50025)
I20240503 08:14:40.701272 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet dfa711e8c8564bc78846b16d68470cea from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:50025: session id = 95860e22c9474e2d894f04deba8442cc-dfa711e8c8564bc78846b16d68470cea
I20240503 08:14:40.705083 14507 tablet_copy_source_session.cc:205] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.706488 16668 raft_consensus.cc:1203] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:40.710886 14340 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:40.712255 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa711e8c8564bc78846b16d68470cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.722980 14751 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:40.724874 18656 tablet_copy_client.cc:783] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.725113 18656 tablet_copy_client.cc:647] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.730736 14489 consensus_queue.cc:237] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:40.737239 18656 tablet_copy_client.cc:523] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:14:40.744881 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cc0fe7d81e149d5b82fabfdc11d66b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.746791 14620 raft_consensus.cc:1261] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.747620 18505 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.756999 18491 ts_tablet_manager.cc:902] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:40.759460 18656 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:40.766147 14751 raft_consensus.cc:1261] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.767134 18505 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.791070 18602 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.795377 14620 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.803120 14340 catalog_manager.cc:5484] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.807646 14751 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:40.832013 18491 tablet_copy_client.cc:308] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:40.832890 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet b19df3ae9f2b44a3818855f259962fff from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:34545)
I20240503 08:14:40.833073 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet b19df3ae9f2b44a3818855f259962fff from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:34545: session id = 036223b0890d4634acfadf9596f39f80-b19df3ae9f2b44a3818855f259962fff
I20240503 08:14:40.836134 14641 tablet_copy_source_session.cc:205] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.839388 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df3ae9f2b44a3818855f259962fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.873275 18491 tablet_copy_client.cc:783] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.873914 18491 tablet_copy_client.cc:647] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.881958 18491 tablet_copy_client.cc:523] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.890080 18491 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:40.919745 18656 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.920492 18656 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:40.920871 18656 ts_tablet_manager.cc:1367] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.171s	user 0.069s	sys 0.000s
I20240503 08:14:40.922072 18656 raft_consensus.cc:348] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:40.922475 18656 raft_consensus.cc:729] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:40.923463 18656 consensus_queue.cc:260] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:40.930677 18656 ts_tablet_manager.cc:1398] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.010s	user 0.000s	sys 0.009s
I20240503 08:14:40.931614 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-dfa711e8c8564bc78846b16d68470cea received from {username='slave'} at 127.2.49.134:50025
I20240503 08:14:40.931764 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-dfa711e8c8564bc78846b16d68470cea on tablet dfa711e8c8564bc78846b16d68470cea with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:40.987169 16404 raft_consensus.cc:1203] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 08:14:40.992373 14687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.129:37829, user_credentials={real_user=slave}} blocked reactor thread for 51866.5us
I20240503 08:14:41.004953 16668 raft_consensus.cc:1203] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:41.006640 14878 consensus_queue.cc:237] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.020082 14751 raft_consensus.cc:1261] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.020979 18090 consensus_queue.cc:1035] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.022653 14489 raft_consensus.cc:1261] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.023452 18090 consensus_queue.cc:1035] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.030637 18491 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.031206 18491 tablet_bootstrap.cc:492] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:41.031522 18491 ts_tablet_manager.cc:1367] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.142s	user 0.058s	sys 0.006s
I20240503 08:14:41.032606 18491 raft_consensus.cc:348] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:41.032868 18491 raft_consensus.cc:729] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:41.033174 18491 consensus_queue.cc:260] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:41.060106 16536 raft_consensus.cc:1203] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:41.073868 18090 raft_consensus.cc:2932] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.086549 14490 raft_consensus.cc:2932] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.097209 18491 ts_tablet_manager.cc:1398] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.066s	user 0.003s	sys 0.024s
I20240503 08:14:41.097991 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-b19df3ae9f2b44a3818855f259962fff received from {username='slave'} at 127.2.49.136:34545
I20240503 08:14:41.098124 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 036223b0890d4634acfadf9596f39f80-b19df3ae9f2b44a3818855f259962fff on tablet b19df3ae9f2b44a3818855f259962fff with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:41.102980 14342 catalog_manager.cc:5484] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.129632 18491 ts_tablet_manager.cc:902] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:41.154337 18491 tablet_copy_client.cc:308] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:41.156740 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 6b677894b51f4dc58f157d8f4d632e9a from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:55303)
I20240503 08:14:41.156934 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 6b677894b51f4dc58f157d8f4d632e9a from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:55303: session id = 036223b0890d4634acfadf9596f39f80-6b677894b51f4dc58f157d8f4d632e9a
I20240503 08:14:41.161458 14507 tablet_copy_source_session.cc:205] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.163251 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b677894b51f4dc58f157d8f4d632e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.165449 14751 consensus_queue.cc:237] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.171602 18491 tablet_copy_client.cc:783] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.171963 18491 tablet_copy_client.cc:647] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.173997 18491 tablet_copy_client.cc:523] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.174079 18776 ts_tablet_manager.cc:902] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:41.179363 18491 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:41.184626 16800 raft_consensus.cc:1203] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 08:14:41.186450 14685 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7aa1031b9e30478597da5ad3708bdff5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:41.189877 14620 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.190867 18433 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.201877 14877 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.202646 18433 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.214563 14751 raft_consensus.cc:2932] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.223100 18776 tablet_copy_client.cc:308] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:41.225534 18433 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.226189 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 562e1ec5c34945298cf457758328057f from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:53581)
I20240503 08:14:41.226357 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 562e1ec5c34945298cf457758328057f from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:53581: session id = 0f8702fad8ea4d22840dab11bb27f7f2-562e1ec5c34945298cf457758328057f
I20240503 08:14:41.230489 14641 tablet_copy_source_session.cc:205] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.233119 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 562e1ec5c34945298cf457758328057f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.237052 14877 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:14:41.241381 14815 consensus_peers.cc:485] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12df78f752c34fe3885b235c4a19e325. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:41.242089 14751 consensus_queue.cc:237] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.245893 14620 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:14:41.247459 14685 consensus_peers.cc:485] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d6c920958214148b3c3d5ac995e99e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:41.247548 14340 catalog_manager.cc:5484] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.249521 14619 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.250097 14489 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.250124 18741 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.250727 18741 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.275779 18809 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.277338 18776 tablet_copy_client.cc:783] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.277724 18776 tablet_copy_client.cc:647] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.279927 18776 tablet_copy_client.cc:523] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.283511 18776 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:41.284615 14619 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.291092 14342 catalog_manager.cc:5484] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.293444 14489 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:41.300262 18491 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.307049 18491 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:41.307731 18491 ts_tablet_manager.cc:1367] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.128s	user 0.058s	sys 0.002s
I20240503 08:14:41.310319 18491 raft_consensus.cc:348] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:41.311300 18491 raft_consensus.cc:729] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:41.311640 18491 consensus_queue.cc:260] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:41.313643 18491 ts_tablet_manager.cc:1398] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 08:14:41.319288 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-6b677894b51f4dc58f157d8f4d632e9a received from {username='slave'} at 127.2.49.136:55303
I20240503 08:14:41.319489 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 036223b0890d4634acfadf9596f39f80-6b677894b51f4dc58f157d8f4d632e9a on tablet 6b677894b51f4dc58f157d8f4d632e9a with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:41.334935 18656 ts_tablet_manager.cc:902] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:41.352581 18656 tablet_copy_client.cc:308] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:41.353603 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:50025)
I20240503 08:14:41.353818 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:50025: session id = 95860e22c9474e2d894f04deba8442cc-7cc0fe7d81e149d5b82fabfdc11d66b4
I20240503 08:14:41.357242 14507 tablet_copy_source_session.cc:205] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.358811 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.372332 18656 tablet_copy_client.cc:783] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.372690 18656 tablet_copy_client.cc:647] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.376346 18656 tablet_copy_client.cc:523] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.382575 18656 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:41.439890 18859 ts_tablet_manager.cc:902] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:41.441059 14751 consensus_queue.cc:237] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
W20240503 08:14:41.445432 14686 consensus_peers.cc:485] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d6c920958214148b3c3d5ac995e99e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:41.446000 18776 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.446343 14489 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:41.446362 14619 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:41.446538 18776 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:41.446826 18776 ts_tablet_manager.cc:1367] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.163s	user 0.051s	sys 0.029s
I20240503 08:14:41.447017 18741 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:41.447293 18433 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:41.447710 18776 raft_consensus.cc:348] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.447995 18776 raft_consensus.cc:729] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:41.448235 18776 consensus_queue.cc:260] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
W20240503 08:14:41.448395 14685 consensus_peers.cc:485] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d6c920958214148b3c3d5ac995e99e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:41.448901 18776 ts_tablet_manager.cc:1398] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:14:41.449753 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-562e1ec5c34945298cf457758328057f received from {username='slave'} at 127.2.49.133:53581
I20240503 08:14:41.449911 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-562e1ec5c34945298cf457758328057f on tablet 562e1ec5c34945298cf457758328057f with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:41.454310 18741 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:41.460731 14619 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:41.475366 14340 catalog_manager.cc:5484] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:41.478843 14489 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:41.488142 18656 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.488796 18656 tablet_bootstrap.cc:492] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:41.489063 18656 ts_tablet_manager.cc:1367] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.107s	user 0.048s	sys 0.009s
I20240503 08:14:41.489873 18656 raft_consensus.cc:348] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:41.490099 18656 raft_consensus.cc:729] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:41.492502 18656 consensus_queue.cc:260] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:41.501237 18656 ts_tablet_manager.cc:1398] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240503 08:14:41.502040 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-7cc0fe7d81e149d5b82fabfdc11d66b4 received from {username='slave'} at 127.2.49.134:50025
I20240503 08:14:41.502197 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-7cc0fe7d81e149d5b82fabfdc11d66b4 on tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:41.572240 18859 tablet_copy_client.cc:308] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:41.627326 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 12df78f752c34fe3885b235c4a19e325 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:60169)
I20240503 08:14:41.627553 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 12df78f752c34fe3885b235c4a19e325 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:60169: session id = 0f8702fad8ea4d22840dab11bb27f7f2-12df78f752c34fe3885b235c4a19e325
I20240503 08:14:41.631130 14901 tablet_copy_source_session.cc:205] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.634086 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12df78f752c34fe3885b235c4a19e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.640882 16404 raft_consensus.cc:1203] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:41.654412 18859 tablet_copy_client.cc:783] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.654733 18859 tablet_copy_client.cc:647] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.679052 16800 raft_consensus.cc:1203] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:41.687580 18859 tablet_copy_client.cc:523] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.691669 18859 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:41.732920 14877 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:41.747126 18776 ts_tablet_manager.cc:902] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
W20240503 08:14:41.765825 14817 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d16c2981213464cbefa44b6ca509182. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:41.767185 14619 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.768146 18760 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.771936 16536 raft_consensus.cc:1203] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:41.776274 14489 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:41.776988 18760 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:41.822697 18491 ts_tablet_manager.cc:902] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:41.872444 18776 tablet_copy_client.cc:308] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:41.873420 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 7aa1031b9e30478597da5ad3708bdff5 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:37499)
I20240503 08:14:41.873622 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 7aa1031b9e30478597da5ad3708bdff5 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:37499: session id = 0f8702fad8ea4d22840dab11bb27f7f2-7aa1031b9e30478597da5ad3708bdff5
I20240503 08:14:41.877933 18859 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.878284 14771 tablet_copy_source_session.cc:205] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.878599 18859 tablet_bootstrap.cc:492] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:41.878963 18859 ts_tablet_manager.cc:1367] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.187s	user 0.063s	sys 0.018s
I20240503 08:14:41.888586 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa1031b9e30478597da5ad3708bdff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.889854 18859 raft_consensus.cc:348] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.890211 18859 raft_consensus.cc:729] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:41.891258 18859 consensus_queue.cc:260] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.891955 18859 ts_tablet_manager.cc:1398] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240503 08:14:41.896584 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-12df78f752c34fe3885b235c4a19e325 received from {username='slave'} at 127.2.49.133:60169
I20240503 08:14:41.896729 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-12df78f752c34fe3885b235c4a19e325 on tablet 12df78f752c34fe3885b235c4a19e325 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:41.900619 18859 ts_tablet_manager.cc:902] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:41.910267 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:41.921924 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:41.923385 18945 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:41.926322 18859 tablet_copy_client.cc:308] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:41.936054 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 1d6c920958214148b3c3d5ac995e99e2 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:37499)
I20240503 08:14:41.936295 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 1d6c920958214148b3c3d5ac995e99e2 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:37499: session id = 0f8702fad8ea4d22840dab11bb27f7f2-1d6c920958214148b3c3d5ac995e99e2
I20240503 08:14:41.940199 14771 tablet_copy_source_session.cc:205] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.941771 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c920958214148b3c3d5ac995e99e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.941932 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:41.949430 14619 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:41.968430 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:41.970340 14342 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:41.976471 18859 tablet_copy_client.cc:783] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.977430 18859 tablet_copy_client.cc:647] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.979354 18859 tablet_copy_client.cc:523] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.981849 18776 tablet_copy_client.cc:783] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.982137 18776 tablet_copy_client.cc:647] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.985612 18776 tablet_copy_client.cc:523] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.992003 18776 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:42.019901 18859 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:42.058773 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:42.061619 14489 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:42.075584 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:42.102310 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:42.124989 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.157415 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:42.160741 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.162242 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.183305 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.229709 18982 ts_tablet_manager.cc:902] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:42.233217 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:42.242365 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.257572 14748 consensus_queue.cc:237] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:42.259382 18859 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:42.259938 18859 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:42.260007 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:42.260263 18859 ts_tablet_manager.cc:1367] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.240s	user 0.045s	sys 0.014s
I20240503 08:14:42.261147 18859 raft_consensus.cc:348] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:42.261407 18859 raft_consensus.cc:729] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:42.261667 18859 consensus_queue.cc:260] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:42.262388 18859 ts_tablet_manager.cc:1398] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:42.265112 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-1d6c920958214148b3c3d5ac995e99e2 received from {username='slave'} at 127.2.49.133:37499
I20240503 08:14:42.265228 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-1d6c920958214148b3c3d5ac995e99e2 on tablet 1d6c920958214148b3c3d5ac995e99e2 with peer 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:42.276234 14685 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.277467 14877 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.278580 18433 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.284441 16404 raft_consensus.cc:1203] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:42.291347 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:42.306233 18491 tablet_copy_client.cc:308] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:42.309952 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 1d6c920958214148b3c3d5ac995e99e2 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:56769)
I20240503 08:14:42.310137 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 1d6c920958214148b3c3d5ac995e99e2 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:56769: session id = 036223b0890d4634acfadf9596f39f80-1d6c920958214148b3c3d5ac995e99e2
I20240503 08:14:42.315485 14771 tablet_copy_source_session.cc:205] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:42.317055 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6c920958214148b3c3d5ac995e99e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:42.325956 18491 tablet_copy_client.cc:783] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:42.326224 18491 tablet_copy_client.cc:647] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
W20240503 08:14:42.327517 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7aa1031b9e30478597da5ad3708bdff5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.329249 18491 tablet_copy_client.cc:523] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:42.332878 18491 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:42.356925 18982 tablet_copy_client.cc:308] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:42.358108 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 5d16c2981213464cbefa44b6ca509182 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:44455)
I20240503 08:14:42.358343 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 5d16c2981213464cbefa44b6ca509182 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:44455: session id = 95860e22c9474e2d894f04deba8442cc-5d16c2981213464cbefa44b6ca509182
I20240503 08:14:42.361387 14901 tablet_copy_source_session.cc:205] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:42.363689 18982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d16c2981213464cbefa44b6ca509182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:42.375177 18982 tablet_copy_client.cc:783] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:42.375711 18982 tablet_copy_client.cc:647] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:42.377899 18982 tablet_copy_client.cc:523] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:42.397156 16404 raft_consensus.cc:1203] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:42.403502 14619 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.413962 18982 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:42.434307 18741 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.451591 18741 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:42.462563 14877 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:42.469050 14342 catalog_manager.cc:5484] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:42.481400 14619 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:42.517295 18776 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:42.517844 18776 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:42.524886 18776 ts_tablet_manager.cc:1367] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.533s	user 0.063s	sys 0.006s
I20240503 08:14:42.525665 18776 raft_consensus.cc:348] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:42.525959 18776 raft_consensus.cc:729] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:42.526283 18776 consensus_queue.cc:260] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:42.529690 18776 ts_tablet_manager.cc:1398] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 08:14:42.533007 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-7aa1031b9e30478597da5ad3708bdff5 received from {username='slave'} at 127.2.49.133:37499
I20240503 08:14:42.533180 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-7aa1031b9e30478597da5ad3708bdff5 on tablet 7aa1031b9e30478597da5ad3708bdff5 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:42.557909 18491 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:42.559504 18491 tablet_bootstrap.cc:492] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:42.559990 18491 ts_tablet_manager.cc:1367] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.227s	user 0.050s	sys 0.004s
I20240503 08:14:42.562088 18491 raft_consensus.cc:348] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:42.562747 18491 raft_consensus.cc:729] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:42.563207 18491 consensus_queue.cc:260] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:42.570521 18491 ts_tablet_manager.cc:1398] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.003s
I20240503 08:14:42.571588 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-1d6c920958214148b3c3d5ac995e99e2 received from {username='slave'} at 127.2.49.136:56769
I20240503 08:14:42.571787 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 036223b0890d4634acfadf9596f39f80-1d6c920958214148b3c3d5ac995e99e2 on tablet 1d6c920958214148b3c3d5ac995e99e2 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:42.620625 18982 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:42.622953 18982 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:42.623817 18982 ts_tablet_manager.cc:1367] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.210s	user 0.050s	sys 0.004s
I20240503 08:14:42.625069 18982 raft_consensus.cc:348] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:42.626408 18982 raft_consensus.cc:729] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:42.627025 18982 consensus_queue.cc:260] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:42.635053 18982 ts_tablet_manager.cc:1398] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.001s
I20240503 08:14:42.636245 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-5d16c2981213464cbefa44b6ca509182 received from {username='slave'} at 127.2.49.134:44455
I20240503 08:14:42.636400 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-5d16c2981213464cbefa44b6ca509182 on tablet 5d16c2981213464cbefa44b6ca509182 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:42.659015 16536 raft_consensus.cc:1203] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:42.741740 14751 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:42.753280 16404 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:42.758781 14620 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.759065 14877 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:42.759814 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f19b7ef418974b06bdcc7125e8cf732d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.761123 18433 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.761834 18433 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.804212 19056 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:42.805409 14878 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:42.806082 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.810621 14342 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:42.812855 14620 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:42.825448 16800 raft_consensus.cc:1203] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:42.854235 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.855749 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:42.887193 19070 ts_tablet_manager.cc:902] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:42.894999 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:42.942754 19070 tablet_copy_client.cc:308] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:42.973114 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:42.989632 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 7aa1031b9e30478597da5ad3708bdff5 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:39075)
I20240503 08:14:42.989831 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 7aa1031b9e30478597da5ad3708bdff5 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:39075: session id = 6835ad958e094ce4927135b942a2e8a8-7aa1031b9e30478597da5ad3708bdff5
I20240503 08:14:42.993063 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:42.993475 14771 tablet_copy_source_session.cc:205] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:42.996323 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa1031b9e30478597da5ad3708bdff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.010250 19070 tablet_copy_client.cc:783] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.010779 19070 tablet_copy_client.cc:647] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.012573 19070 tablet_copy_client.cc:523] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.014310 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:43.037798 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:43.040351 19070 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:43.225525 19070 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.226140 19070 tablet_bootstrap.cc:492] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:43.226529 19070 ts_tablet_manager.cc:1367] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.186s	user 0.049s	sys 0.013s
I20240503 08:14:43.227623 19070 raft_consensus.cc:348] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:43.227960 19070 raft_consensus.cc:729] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:43.228250 19070 consensus_queue.cc:260] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:43.233701 19131 ts_tablet_manager.cc:902] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:43.252740 14878 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:43.255604 14877 consensus_queue.cc:237] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
W20240503 08:14:43.256292 14815 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d16c2981213464cbefa44b6ca509182. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.257246 14490 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.257922 18945 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.259074 19131 tablet_copy_client.cc:308] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:43.260335 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet f19b7ef418974b06bdcc7125e8cf732d from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:44645)
I20240503 08:14:43.260522 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet f19b7ef418974b06bdcc7125e8cf732d from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:44645: session id = 95860e22c9474e2d894f04deba8442cc-f19b7ef418974b06bdcc7125e8cf732d
I20240503 08:14:43.264035 16536 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.265069 14620 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:43.266968 14816 consensus_peers.cc:485] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0780fb979eee4bae98a014d9721c9441. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.267246 18945 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.267874 18945 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.271728 19070 ts_tablet_manager.cc:1398] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.008s
I20240503 08:14:43.273870 14770 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-7aa1031b9e30478597da5ad3708bdff5 received from {username='slave'} at 127.2.49.135:39075
I20240503 08:14:43.273945 14619 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.274047 14770 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-7aa1031b9e30478597da5ad3708bdff5 on tablet 7aa1031b9e30478597da5ad3708bdff5 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:43.276752 19151 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.281880 14748 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.282743 14771 tablet_copy_source_session.cc:205] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.285882 19131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19b7ef418974b06bdcc7125e8cf732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.286243 18945 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.295389 19131 tablet_copy_client.cc:783] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.295667 19131 tablet_copy_client.cc:647] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.320564 19131 tablet_copy_client.cc:523] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.334637 18945 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:43.339682 14489 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:43.348826 19151 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.349242 14348 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:43.354600 14621 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.359202 19131 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:43.363569 16536 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:43.386363 14748 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.396989 14340 catalog_manager.cc:5484] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index -1 to 4, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.408059 14621 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:43.596590 19131 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.597183 19131 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:43.597564 19131 ts_tablet_manager.cc:1367] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.238s	user 0.055s	sys 0.024s
I20240503 08:14:43.605093 19131 raft_consensus.cc:348] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:43.605454 19131 raft_consensus.cc:729] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:43.606626 19131 consensus_queue.cc:260] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:43.607854 19131 ts_tablet_manager.cc:1398] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.001s
I20240503 08:14:43.608742 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-f19b7ef418974b06bdcc7125e8cf732d received from {username='slave'} at 127.2.49.134:44645
I20240503 08:14:43.608914 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-f19b7ef418974b06bdcc7125e8cf732d on tablet f19b7ef418974b06bdcc7125e8cf732d with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:43.673774 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:43.691960 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:43.707616 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:43.724665 16668 raft_consensus.cc:1203] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:43.737543 16536 raft_consensus.cc:1203] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:43.740523 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:43.756062 19253 ts_tablet_manager.cc:902] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:43.764524 19253 tablet_copy_client.cc:308] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:43.766068 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 5d16c2981213464cbefa44b6ca509182 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:60169)
I20240503 08:14:43.766290 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 5d16c2981213464cbefa44b6ca509182 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:60169: session id = 0f8702fad8ea4d22840dab11bb27f7f2-5d16c2981213464cbefa44b6ca509182
I20240503 08:14:43.769119 14901 tablet_copy_source_session.cc:205] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.771099 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:43.771636 19253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d16c2981213464cbefa44b6ca509182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.789284 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:43.790958 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:43.801055 14490 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:43.810516 14748 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
W20240503 08:14:43.815157 14686 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f19b7ef418974b06bdcc7125e8cf732d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.820047 19253 tablet_copy_client.cc:783] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.823683 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:43.825604 14877 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.826428 18433 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.826915 16536 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.827538 18433 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.829978 14621 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.838856 18433 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.839248 19253 tablet_copy_client.cc:647] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.861270 14877 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.862998 18139 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.863214 16801 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.863948 18139 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.865067 14621 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.866912 18139 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.867439 19253 tablet_copy_client.cc:523] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:14:43.869621 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b677894b51f4dc58f157d8f4d632e9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.870699 19275 ts_tablet_manager.cc:902] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:43.871152 18139 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:43.874986 14621 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:43.880105 19273 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.883695 16801 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:43.884264 14345 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:43.925141 19275 tablet_copy_client.cc:308] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:43.928110 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 0780fb979eee4bae98a014d9721c9441 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:41491)
I20240503 08:14:43.928326 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 0780fb979eee4bae98a014d9721c9441 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:41491: session id = 036223b0890d4634acfadf9596f39f80-0780fb979eee4bae98a014d9721c9441
I20240503 08:14:43.932698 14901 tablet_copy_source_session.cc:205] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.934607 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0780fb979eee4bae98a014d9721c9441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.934283 14621 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.938355 16536 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.944981 14877 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:43.951413 19275 tablet_copy_client.cc:783] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.952047 19275 tablet_copy_client.cc:647] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.953019 14345 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:43.959587 19275 tablet_copy_client.cc:523] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.966109 19275 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:14:44.006336 19253 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:44.021404 14877 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:44.063536 19275 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:44.090551 19275 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:44.090903 19275 ts_tablet_manager.cc:1367] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.126s	user 0.051s	sys 0.006s
I20240503 08:14:44.091801 19275 raft_consensus.cc:348] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:44.092053 19275 raft_consensus.cc:729] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:44.092296 19275 consensus_queue.cc:260] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:44.092864 19275 ts_tablet_manager.cc:1398] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:14:44.093930 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-0780fb979eee4bae98a014d9721c9441 received from {username='slave'} at 127.2.49.136:41491
I20240503 08:14:44.094084 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 036223b0890d4634acfadf9596f39f80-0780fb979eee4bae98a014d9721c9441 on tablet 0780fb979eee4bae98a014d9721c9441 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:44.188661 19253 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:44.189613 19253 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:44.190151 19253 ts_tablet_manager.cc:1367] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.184s	user 0.067s	sys 0.006s
I20240503 08:14:44.191882 19253 raft_consensus.cc:348] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:44.192358 19253 raft_consensus.cc:729] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:44.192701 19253 consensus_queue.cc:260] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:44.259370 19253 ts_tablet_manager.cc:1398] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.069s	user 0.018s	sys 0.027s
I20240503 08:14:44.263741 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-5d16c2981213464cbefa44b6ca509182 received from {username='slave'} at 127.2.49.133:60169
I20240503 08:14:44.263959 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-5d16c2981213464cbefa44b6ca509182 on tablet 5d16c2981213464cbefa44b6ca509182 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:44.265363 14621 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:44.278915 14748 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.279294 16405 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.279542 19101 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:44.280396 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 562e1ec5c34945298cf457758328057f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:44.281318 19101 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.283866 14877 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.284497 19101 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.306818 19275 ts_tablet_manager.cc:902] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:14:44.340821 19101 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:44.352691 14748 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:44.363287 14345 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:44.363968 16405 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:44.452507 19275 tablet_copy_client.cc:308] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:14:44.453233 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet f19b7ef418974b06bdcc7125e8cf732d from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:56769)
I20240503 08:14:44.453430 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet f19b7ef418974b06bdcc7125e8cf732d from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:56769: session id = 036223b0890d4634acfadf9596f39f80-f19b7ef418974b06bdcc7125e8cf732d
I20240503 08:14:44.456948 14771 tablet_copy_source_session.cc:205] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:44.458528 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19b7ef418974b06bdcc7125e8cf732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:44.471983 14877 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:44.477490 16801 raft_consensus.cc:1203] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:14:44.484359 19372 ts_tablet_manager.cc:902] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:44.492558 19275 tablet_copy_client.cc:783] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:44.492933 19275 tablet_copy_client.cc:647] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:44.494856 19275 tablet_copy_client.cc:523] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:44.501648 19275 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
W20240503 08:14:44.512176 14814 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 185527us
I20240503 08:14:44.556090 19372 tablet_copy_client.cc:308] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:44.556951 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 6b677894b51f4dc58f157d8f4d632e9a from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:50025)
I20240503 08:14:44.557111 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 6b677894b51f4dc58f157d8f4d632e9a from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:50025: session id = 95860e22c9474e2d894f04deba8442cc-6b677894b51f4dc58f157d8f4d632e9a
I20240503 08:14:44.560396 14507 tablet_copy_source_session.cc:205] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:44.565459 19372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b677894b51f4dc58f157d8f4d632e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:44.572193 19372 tablet_copy_client.cc:783] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:44.572456 19372 tablet_copy_client.cc:647] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:44.579870 19372 tablet_copy_client.cc:523] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:44.591792 19372 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:14:44.605937 16405 raft_consensus.cc:1203] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:44.652798 19275 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:44.653442 19275 tablet_bootstrap.cc:492] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:14:44.653774 19275 ts_tablet_manager.cc:1367] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.152s	user 0.056s	sys 0.001s
I20240503 08:14:44.654850 19275 raft_consensus.cc:348] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:44.655201 19275 raft_consensus.cc:729] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:14:44.655567 19275 consensus_queue.cc:260] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:44.660396 19275 ts_tablet_manager.cc:1398] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240503 08:14:44.665263 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-f19b7ef418974b06bdcc7125e8cf732d received from {username='slave'} at 127.2.49.136:56769
I20240503 08:14:44.665408 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 036223b0890d4634acfadf9596f39f80-f19b7ef418974b06bdcc7125e8cf732d on tablet f19b7ef418974b06bdcc7125e8cf732d with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:14:44.702338 16801 raft_consensus.cc:1203] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:44.750218 19446 ts_tablet_manager.cc:902] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:44.759047 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:44.779810 19372 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:44.780470 19372 tablet_bootstrap.cc:492] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:14:44.780834 19372 ts_tablet_manager.cc:1367] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.189s	user 0.078s	sys 0.016s
I20240503 08:14:44.781713 19372 raft_consensus.cc:348] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:44.782239 19372 raft_consensus.cc:729] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:14:44.782553 19372 consensus_queue.cc:260] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:44.786461 19446 tablet_copy_client.cc:308] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:14:44.787254 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 562e1ec5c34945298cf457758328057f from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:57463)
I20240503 08:14:44.787472 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 562e1ec5c34945298cf457758328057f from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:57463: session id = 6835ad958e094ce4927135b942a2e8a8-562e1ec5c34945298cf457758328057f
I20240503 08:14:44.790372 19372 ts_tablet_manager.cc:1398] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20240503 08:14:44.791504 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-6b677894b51f4dc58f157d8f4d632e9a received from {username='slave'} at 127.2.49.134:50025
I20240503 08:14:44.791683 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-6b677894b51f4dc58f157d8f4d632e9a on tablet 6b677894b51f4dc58f157d8f4d632e9a with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:44.791755 14641 tablet_copy_source_session.cc:205] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:44.793382 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 562e1ec5c34945298cf457758328057f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:44.804085 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:44.814749 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:44.851034 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:44.884127 16536 raft_consensus.cc:1203] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:44.904711 19446 tablet_copy_client.cc:783] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:44.905069 19446 tablet_copy_client.cc:647] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:44.908280 19446 tablet_copy_client.cc:523] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:44.921836 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:44.931897 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:44.955590 19446 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:45.025334 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:45.055140 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.126643 19446 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:45.150496 19446 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:45.150882 19446 ts_tablet_manager.cc:1367] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.195s	user 0.050s	sys 0.009s
I20240503 08:14:45.151834 19446 raft_consensus.cc:348] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:45.152092 19446 raft_consensus.cc:729] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:45.152345 19446 consensus_queue.cc:260] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:14:45.226276 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 69458.4us
I20240503 08:14:45.227658 19446 ts_tablet_manager.cc:1398] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.077s	user 0.012s	sys 0.023s
I20240503 08:14:45.228709 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-562e1ec5c34945298cf457758328057f received from {username='slave'} at 127.2.49.135:57463
I20240503 08:14:45.228876 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-562e1ec5c34945298cf457758328057f on tablet 562e1ec5c34945298cf457758328057f with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:45.392174 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.425041 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:45.450605 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.464407 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:45.478636 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.479104 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.511818 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:45.582060 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.627067 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:45.652685 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.690191 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:45.691829 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.704391 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:45.732717 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.749941 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:45.783282 16668 raft_consensus.cc:1203] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:14:45.895527 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.922763 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:45.956979 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:45.963749 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:45.996477 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:46.020779 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:14:46.063722 14817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:45663, user_credentials={real_user=slave}} blocked reactor thread for 59215.7us
I20240503 08:14:46.076007 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:46.112061 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:46.113677 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:46.115264 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:46.129885 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:46.158974 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:46.169731 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:46.184875 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:46.195430 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:46.198088 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:46.267118 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:46.658807 14490 consensus_queue.cc:237] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:46.664484 14621 raft_consensus.cc:1261] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.664783 16536 raft_consensus.cc:1261] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.665120 18139 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.665441 19271 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.679348 19271 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:46.688169 16536 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:46.689481 14343 catalog_manager.cc:5484] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:46.701640 14619 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:46.719969 14751 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:46.739964 16801 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.740893 19500 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.752120 14490 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.753171 19500 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.775141 19500 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:46.779551 14490 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:46.780282 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa711e8c8564bc78846b16d68470cea with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:46.800520 16801 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:46.808801 14344 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:46.815230 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet c130eb7ea8a74f4b8d9b79e23af70067 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48320
I20240503 08:14:46.848903 19712 tablet_replica.cc:330] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:14:46.850970 19712 raft_consensus.cc:2229] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:46.858430 19712 raft_consensus.cc:2258] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:46.860884 19712 ts_tablet_manager.cc:1875] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:46.884487 19700 tablet_replica.cc:330] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:46.888839 19712 ts_tablet_manager.cc:1888] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:46.897598 19700 raft_consensus.cc:2229] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:46.898147 19700 raft_consensus.cc:2258] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:46.897977 19712 log.cc:1196] T c130eb7ea8a74f4b8d9b79e23af70067 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/c130eb7ea8a74f4b8d9b79e23af70067
I20240503 08:14:46.899605 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet c130eb7ea8a74f4b8d9b79e23af70067 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:14:46.900771 19700 ts_tablet_manager.cc:1875] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:46.913796 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:46.952085 14621 consensus_queue.cc:237] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:46.953845 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:46.954612 16801 raft_consensus.cc:1261] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.955209 19710 consensus_queue.cc:1035] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.958312 14878 raft_consensus.cc:1261] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.958937 19710 consensus_queue.cc:1035] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.974233 19291 raft_consensus.cc:2932] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:46.975060 16801 raft_consensus.cc:2932] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:46.981606 14344 catalog_manager.cc:5484] T b19df3ae9f2b44a3818855f259962fff P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:47.004580 14878 raft_consensus.cc:2932] T b19df3ae9f2b44a3818855f259962fff P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:47.006752 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet b19df3ae9f2b44a3818855f259962fff with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45478
I20240503 08:14:47.044823 14469 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:47.050526 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.062578 19749 tablet_replica.cc:330] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:14:47.063133 19749 raft_consensus.cc:2229] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:47.063633 19749 raft_consensus.cc:2258] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:47.065310 19749 ts_tablet_manager.cc:1875] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:47.073459 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:47.085088 19700 ts_tablet_manager.cc:1888] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:47.085285 19700 log.cc:1196] T dfa711e8c8564bc78846b16d68470cea P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/dfa711e8c8564bc78846b16d68470cea
I20240503 08:14:47.087581 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet dfa711e8c8564bc78846b16d68470cea (table rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]) successfully deleted
I20240503 08:14:47.094084 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.109699 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:47.126579 19749 ts_tablet_manager.cc:1888] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:47.126777 19749 log.cc:1196] T b19df3ae9f2b44a3818855f259962fff P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/b19df3ae9f2b44a3818855f259962fff
I20240503 08:14:47.128211 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet b19df3ae9f2b44a3818855f259962fff (table rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]) successfully deleted
I20240503 08:14:47.151252 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.645169 14619 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:47.679328 16536 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.680016 19710 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.682267 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.689988 14490 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.690724 19710 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.699461 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:47.712685 19710 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:47.714717 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.722498 14489 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:47.727424 14342 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:47.735230 16536 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:47.747798 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.767735 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:47.772181 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:47.841904 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.846803 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:47.858469 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:47.862073 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:47.885310 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c595bb19d6946b0b1f853064892e946 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:47.913080 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.920331 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:47.921550 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.941915 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.943020 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:47.955744 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:47.988221 19908 tablet_replica.cc:330] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:47.988477 19908 raft_consensus.cc:2229] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:47.988737 19908 raft_consensus.cc:2258] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:47.989495 19908 ts_tablet_manager.cc:1875] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:48.002594 16517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.002713 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.012940 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:48.013491 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:14:48.013947 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 106149us
I20240503 08:14:48.036561 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:48.058557 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.069679 19908 ts_tablet_manager.cc:1888] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:48.069837 19908 log.cc:1196] T 5c595bb19d6946b0b1f853064892e946 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c595bb19d6946b0b1f853064892e946
I20240503 08:14:48.075320 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 5c595bb19d6946b0b1f853064892e946 (table rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]) successfully deleted
I20240503 08:14:48.096989 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:48.130904 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.476974 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.486303 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:48.556692 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.559689 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.599737 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:48.680160 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:48.701826 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:48.725980 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.933564 14621 consensus_queue.cc:237] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:14:48.939178 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 117568us
I20240503 08:14:48.946041 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:48.967098 14877 raft_consensus.cc:1261] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.967839 19710 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.970136 16668 raft_consensus.cc:1261] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.971310 19291 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.986435 19710 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:48.992424 14877 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:48.997709 14343 catalog_manager.cc:5484] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:49.012908 16669 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:49.037868 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:14:49.039523 14557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:45663, user_credentials={real_user=slave}} blocked reactor thread for 48411.9us
I20240503 08:14:49.040810 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.047959 14468 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:14:49.089838 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.090755 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.119092 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:49.172271 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:14:49.173453 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a216295a8a4feaa2d3326035879073 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45478
I20240503 08:14:49.200157 20059 tablet_replica.cc:330] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:14:49.200906 20059 raft_consensus.cc:2229] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.202425 20059 raft_consensus.cc:2258] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.203346 20059 ts_tablet_manager.cc:1875] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:49.218559 20059 ts_tablet_manager.cc:1888] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:49.221669 20059 log.cc:1196] T 76a216295a8a4feaa2d3326035879073 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/76a216295a8a4feaa2d3326035879073
I20240503 08:14:49.222765 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 76a216295a8a4feaa2d3326035879073 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:14:49.275991 14877 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:49.314118 16801 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:49.316339 20051 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:14:49.326887 14621 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:49.327811 19602 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:49.349251 14489 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46184
I20240503 08:14:49.349557 14489 raft_consensus.cc:595] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:49.363989 20051 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:49.366955 14619 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:49.389431 14344 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:49.396957 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 88404b9cdf574cb0bca1bb49348fb4cd with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:49.401979 16800 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:49.421648 20093 tablet_replica.cc:330] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:49.422137 20093 raft_consensus.cc:2229] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.423359 20093 raft_consensus.cc:2258] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.427631 20093 ts_tablet_manager.cc:1875] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:49.440559 19842 raft_consensus.cc:979] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: : Instructing follower c0fd1ebec9da48d4b0c1ae0f0ee96452 to start an election
I20240503 08:14:49.440737 19842 raft_consensus.cc:1067] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Signalling peer c0fd1ebec9da48d4b0c1ae0f0ee96452 to start an election
I20240503 08:14:49.442044 14749 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a"
dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
 from {username='slave'} at 127.2.49.129:59689
I20240503 08:14:49.442396 14749 raft_consensus.cc:482] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:49.442543 14749 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:49.447923 14749 raft_consensus.cc:504] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:49.449450 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:49.449697 14489 raft_consensus.cc:3032] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:49.449855 14489 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:49.450199 14489 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:49.450727 14489 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:49.453693 14489 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 2.
I20240503 08:14:49.455479 14749 leader_election.cc:288] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 2 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075), 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:14:49.457320 16669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6835ad958e094ce4927135b942a2e8a8"
I20240503 08:14:49.457751 16669 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:49.461735 14877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:14:49.462136 14877 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:49.467600 16669 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 2.
I20240503 08:14:49.478552 14877 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd1ebec9da48d4b0c1ae0f0ee96452 in term 2.
I20240503 08:14:49.479318 14685 leader_election.cc:302] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6835ad958e094ce4927135b942a2e8a8, b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:49.479792 20016 raft_consensus.cc:2781] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:49.479980 20016 raft_consensus.cc:686] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 LEADER]: Becoming Leader. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:14:49.480397 20016 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:49.499534 14342 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: term changed from 1 to 2, leader changed from 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) to c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:49.597712 20093 ts_tablet_manager.cc:1888] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:49.615911 20093 log.cc:1196] T 88404b9cdf574cb0bca1bb49348fb4cd P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/88404b9cdf574cb0bca1bb49348fb4cd
I20240503 08:14:49.621914 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 88404b9cdf574cb0bca1bb49348fb4cd (table rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]) successfully deleted
I20240503 08:14:49.626150 14489 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46196
I20240503 08:14:49.626492 14489 raft_consensus.cc:595] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 08:14:49.626657 14489 raft_consensus.cc:603] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 08:14:49.778358 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.811821 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:49.839140 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.851469 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:49.899919 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.916297 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:49.953724 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:49.962574 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:49.977355 14489 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:14:49.978147 20004 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:50.066857 16669 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:14:50.067763 20004 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:50.105772 14877 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:14:50.106581 20016 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:50.295341 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.305336 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:50.376950 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.425031 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:50.479367 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.479550 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.500602 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.518879 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:50.545472 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:50.546491 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:50.549938 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.620592 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.624991 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.671932 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:50.701457 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:50.704327 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:50.714140 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:50.743592 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.821763 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.854665 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:50.860103 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:50.879179 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.887555 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.910038 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:50.937331 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:50.937870 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.943946 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.995517 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:51.005007 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:51.008378 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:51.017352 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:51.032708 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:51.036630 14490 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:51.051936 14751 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:51.052596 19842 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:51.069201 16405 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:51.069869 19842 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:51.092648 20117 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:51.097934 16405 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:51.127781 14344 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:51.131456 14751 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:51.194340 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet a87c5af03f4d47d38bbfbd225c7195d1 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48320
I20240503 08:14:51.216292 20332 tablet_replica.cc:330] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:14:51.216599 20332 raft_consensus.cc:2229] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:51.216892 20332 raft_consensus.cc:2258] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:51.217779 20332 ts_tablet_manager.cc:1875] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:51.226038 20332 ts_tablet_manager.cc:1888] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:51.226200 20332 log.cc:1196] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/a87c5af03f4d47d38bbfbd225c7195d1
I20240503 08:14:51.227118 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet a87c5af03f4d47d38bbfbd225c7195d1 (table rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]) successfully deleted
I20240503 08:14:51.339399 14490 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:51.347723 16800 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:51.347999 16537 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:51.348513 14621 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:51.348445 19842 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:51.349076 20117 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:51.349386 19842 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:51.388803 19842 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.398244 16537 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.399295 16800 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.402066 14343 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.406998 14621 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.410070 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fea213e5a12452f8b894b9cdb94f9ae with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48320
I20240503 08:14:51.410459 20332 tablet_replica.cc:330] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:14:51.410708 20332 raft_consensus.cc:2229] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:51.411058 20332 raft_consensus.cc:2258] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:51.411932 20332 ts_tablet_manager.cc:1875] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:51.427774 20332 ts_tablet_manager.cc:1888] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:51.427958 20332 log.cc:1196] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/3fea213e5a12452f8b894b9cdb94f9ae
I20240503 08:14:51.428824 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 3fea213e5a12452f8b894b9cdb94f9ae (table rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]) successfully deleted
I20240503 08:14:51.445243 14751 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:51.454938 16800 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:51.455096 14490 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:51.455585 20206 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:51.456182 20206 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:51.471163 20377 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.474992 16800 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.481617 14343 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.493574 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 897a701b880641b08816f2d1a254ef03 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48320
I20240503 08:14:51.493984 20332 tablet_replica.cc:330] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:14:51.494217 20332 raft_consensus.cc:2229] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:51.494583 20332 raft_consensus.cc:2258] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:51.495285 20332 ts_tablet_manager.cc:1875] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:51.508383 14490 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:51.526974 20332 ts_tablet_manager.cc:1888] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:51.527170 20332 log.cc:1196] T 897a701b880641b08816f2d1a254ef03 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/897a701b880641b08816f2d1a254ef03
I20240503 08:14:51.541280 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 897a701b880641b08816f2d1a254ef03 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:14:51.697904 14751 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:51.721817 16668 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:51.722623 20377 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:51.723557 14878 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:51.724115 20377 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:51.741641 20377 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:51.744292 16668 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:51.749207 14877 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:51.755005 14347 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:51.760840 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9575f64eb741948e51ccf06f48228a with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45478
I20240503 08:14:51.761953 20473 tablet_replica.cc:330] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:14:51.762269 20473 raft_consensus.cc:2229] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:51.763338 20473 raft_consensus.cc:2258] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:51.769552 20473 ts_tablet_manager.cc:1875] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:51.787678 20473 ts_tablet_manager.cc:1888] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:14:51.787868 20473 log.cc:1196] T dc9575f64eb741948e51ccf06f48228a P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc9575f64eb741948e51ccf06f48228a
I20240503 08:14:51.788744 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet dc9575f64eb741948e51ccf06f48228a (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:14:51.800287 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:51.853256 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:51.912470 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:51.979353 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:52.016239 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.017328 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.033763 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:52.069007 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:52.284966 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.311789 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:52.340551 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.342831 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:52.366602 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.460001 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.490722 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:52.545285 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:52.592809 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.612803 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:52.675352 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.692204 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:52.698839 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.725348 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:52.768389 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:14:52.800316 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 63914.3us
W20240503 08:14:52.814921 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 76536.2us
I20240503 08:14:52.868623 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:14:52.882182 14684 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 85124.7us
I20240503 08:14:52.970871 14490 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:52.974030 16668 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:52.974632 14621 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:52.974740 20529 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:52.975502 20529 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:53.024286 14619 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.024187 20529 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.031550 14343 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.079880 16669 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.114367 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:53.116775 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:53.135282 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.173815 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.187440 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:53.188414 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.194257 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.222208 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.222992 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:53.231551 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b13cdf4b234f17872f5d1588915968 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48320
I20240503 08:14:53.250934 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:53.252434 20679 tablet_replica.cc:330] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:14:53.253675 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.255081 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.261406 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:53.272341 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:53.286607 20679 raft_consensus.cc:2229] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.287029 20679 raft_consensus.cc:2258] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.288022 20679 ts_tablet_manager.cc:1875] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.291735 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:53.317333 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:53.341835 20679 ts_tablet_manager.cc:1888] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:53.371585 20679 log.cc:1196] T 30b13cdf4b234f17872f5d1588915968 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/30b13cdf4b234f17872f5d1588915968
I20240503 08:14:53.386358 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 30b13cdf4b234f17872f5d1588915968 (table rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]) successfully deleted
I20240503 08:14:53.592890 14751 consensus_queue.cc:237] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:53.624799 16537 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.626307 20613 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.676986 16405 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.683877 20727 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:14:53.691366 14687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:36937, user_credentials={real_user=slave}} blocked reactor thread for 65686.2us
I20240503 08:14:53.695801 14619 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:53.708573 16667 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.709663 16405 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.709781 14878 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.710613 20643 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.712095 14877 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.713388 20299 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.714025 20643 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.722441 20727 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.727784 20727 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:53.754751 16537 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:53.765987 16405 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:53.773895 14880 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:53.775410 14343 catalog_manager.cc:5484] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
W20240503 08:14:53.791760 20745 log.cc:924] Time spent T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240503 08:14:53.800770 20742 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.811427 16667 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.817512 14490 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:53.819213 14878 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:53.822705 14880 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.834064 16405 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.834846 20682 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.843731 14348 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.844846 14489 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.845546 20682 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.847692 16536 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:53.846196 16404 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:53.849043 20565 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:53.871898 20565 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.872574 16404 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.875870 16405 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:53.876585 20690 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:53.880952 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ceefc466f642d08ea96d549a534776 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:53.896061 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 562e1ec5c34945298cf457758328057f with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:53.897647 20790 tablet_replica.cc:330] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:53.897923 20790 raft_consensus.cc:2229] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.898511 20790 raft_consensus.cc:2258] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.900383 14341 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.901297 20790 ts_tablet_manager.cc:1875] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.904474 14600 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d16c2981213464cbefa44b6ca509182 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:53.912757 14880 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:53.913663 20690 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:53.914072 16537 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.924005 20690 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.927759 16403 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.929033 20790 ts_tablet_manager.cc:1888] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:53.929275 20790 log.cc:1196] T 562e1ec5c34945298cf457758328057f P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/562e1ec5c34945298cf457758328057f
I20240503 08:14:53.933210 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 562e1ec5c34945298cf457758328057f (table rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]) successfully deleted
I20240503 08:14:53.937041 14341 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.949153 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a949e76ab5477680a4f9912e9672e9 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:53.960882 20790 tablet_replica.cc:330] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:53.961159 20790 raft_consensus.cc:2229] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.961532 20790 raft_consensus.cc:2258] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.962461 20790 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.986083 14490 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:53.995965 20787 tablet_replica.cc:330] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:53.996486 20787 raft_consensus.cc:2229] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.996811 20787 raft_consensus.cc:2258] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:54.001747 14880 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:54.015861 20787 ts_tablet_manager.cc:1875] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:54.024113 20787 ts_tablet_manager.cc:1888] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:54.024403 20787 log.cc:1196] T b5ceefc466f642d08ea96d549a534776 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/b5ceefc466f642d08ea96d549a534776
I20240503 08:14:54.026067 20790 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:54.026239 20790 log.cc:1196] T 72a949e76ab5477680a4f9912e9672e9 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/72a949e76ab5477680a4f9912e9672e9
I20240503 08:14:54.026836 20787 tablet_replica.cc:330] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:54.027105 20787 raft_consensus.cc:2229] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:54.027324 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 72a949e76ab5477680a4f9912e9672e9 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:14:54.027464 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet b5ceefc466f642d08ea96d549a534776 (table rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]) successfully deleted
I20240503 08:14:54.028465 20787 raft_consensus.cc:2258] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:54.029597 20787 ts_tablet_manager.cc:1875] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:54.045946 20787 ts_tablet_manager.cc:1888] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:54.046828 20787 log.cc:1196] T 5d16c2981213464cbefa44b6ca509182 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d16c2981213464cbefa44b6ca509182
I20240503 08:14:54.048296 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 5d16c2981213464cbefa44b6ca509182 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:14:54.184193 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:54.192835 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.239477 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.245627 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:54.256029 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.278223 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:54.278649 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.285244 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:54.616797 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.674188 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.688082 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:54.698446 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.728525 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.736801 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:54.741480 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:54.836135 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:14:54.911126 14556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 61293.8us
I20240503 08:14:54.944948 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.945067 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:55.038714 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.041213 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.052233 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:55.059528 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:55.064337 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.082283 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.096118 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.097648 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.103596 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:55.106494 14751 consensus_queue.cc:237] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:55.113348 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:55.117810 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.137953 16667 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:55.138183 16403 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:55.138762 20737 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:55.139765 20737 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:55.142145 14880 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:55.142907 20737 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:55.152675 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.176918 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:55.192552 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:55.197916 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.200759 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.260038 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:55.262419 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:55.279790 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:55.289948 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:55.297798 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:55.304622 20948 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:55.310532 14878 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:55.314283 16667 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:55.323102 14342 catalog_manager.cc:5484] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:55.329910 16403 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:55.401795 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:55.417066 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa1031b9e30478597da5ad3708bdff5 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:55.430219 21018 tablet_replica.cc:330] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:55.430588 21018 raft_consensus.cc:2229] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:55.430904 21018 raft_consensus.cc:2258] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:55.434019 21018 ts_tablet_manager.cc:1875] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:55.454792 21018 ts_tablet_manager.cc:1888] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:55.455039 21018 log.cc:1196] T 7aa1031b9e30478597da5ad3708bdff5 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/7aa1031b9e30478597da5ad3708bdff5
I20240503 08:14:55.455941 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 7aa1031b9e30478597da5ad3708bdff5 (table rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]) successfully deleted
I20240503 08:14:55.701929 14490 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:55.740891 16800 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:55.740991 16537 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:55.741605 20690 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:55.742125 19842 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:55.742444 14621 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:55.744647 20690 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:55.760692 19842 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:55.764175 16800 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:55.777137 16537 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:55.779523 14347 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:55.784983 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b677894b51f4dc58f157d8f4d632e9a with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48320
I20240503 08:14:55.827917 21063 tablet_replica.cc:330] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:14:55.840704 21063 raft_consensus.cc:2229] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:55.841302 21063 raft_consensus.cc:2258] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:55.842396 21063 ts_tablet_manager.cc:1875] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:55.855926 14619 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:55.909169 21063 ts_tablet_manager.cc:1888] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:55.909468 21063 log.cc:1196] T 6b677894b51f4dc58f157d8f4d632e9a P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/6b677894b51f4dc58f157d8f4d632e9a
I20240503 08:14:55.910614 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 6b677894b51f4dc58f157d8f4d632e9a (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:14:55.985826 14490 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:55.998924 16403 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:55.999475 21061 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:56.002079 14621 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:56.002723 20690 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:56.002642 21061 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:56.006551 16403 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:56.015655 14347 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:56.022841 14621 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:56.062422 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet f65ab6dba1124573962e2da58aa1859c with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:56.068010 21127 tablet_replica.cc:330] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:56.068224 21127 raft_consensus.cc:2229] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:56.068444 21127 raft_consensus.cc:2258] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:56.069238 21127 ts_tablet_manager.cc:1875] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:56.076581 21127 ts_tablet_manager.cc:1888] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:56.076714 21127 log.cc:1196] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/f65ab6dba1124573962e2da58aa1859c
I20240503 08:14:56.077647 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet f65ab6dba1124573962e2da58aa1859c (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:14:56.091740 14749 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:56.101603 16800 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:56.102391 21003 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:56.116232 16537 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:56.116848 21003 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:56.127655 14878 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:56.128381 21003 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:56.177065 20948 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:56.185950 16536 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:56.216652 14348 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:56.225095 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet f19b7ef418974b06bdcc7125e8cf732d with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:56.226563 14878 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:56.233381 16800 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:56.239015 21146 tablet_replica.cc:330] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:56.239238 21146 raft_consensus.cc:2229] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:56.239488 21146 raft_consensus.cc:2258] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:56.243095 21146 ts_tablet_manager.cc:1875] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:56.261307 21146 ts_tablet_manager.cc:1888] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:56.262569 21146 log.cc:1196] T f19b7ef418974b06bdcc7125e8cf732d P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/f19b7ef418974b06bdcc7125e8cf732d
I20240503 08:14:56.263646 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet f19b7ef418974b06bdcc7125e8cf732d (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:14:56.413841 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.499696 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:56.569490 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.581718 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:56.632325 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.644403 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:56.651374 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:56.660009 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.686664 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.703406 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:56.708249 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:56.745047 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.749815 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.769716 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:56.852411 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:56.977922 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:57.061419 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.061837 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.084141 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:57.089327 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:57.106626 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.136294 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.205633 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:57.228036 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:57.355778 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:57.422890 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.427058 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:57.443765 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.465728 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.480311 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.517472 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:57.558979 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:57.589143 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.590309 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:57.613947 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:57.662513 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.663748 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.672411 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.700886 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:57.728143 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:57.769979 14751 consensus_queue.cc:237] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:57.791414 14880 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:57.805024 14490 consensus_queue.cc:237] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:57.850924 14749 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:57.851114 16667 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:57.851482 16537 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:57.851686 21263 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:57.852878 21264 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:57.853593 21264 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:57.860688 16800 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:57.861610 21141 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:57.873303 16403 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:57.873301 14489 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:57.875213 20948 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:57.876111 20948 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:57.879652 14621 raft_consensus.cc:1261] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:57.880784 21061 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:57.886662 16536 raft_consensus.cc:1261] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:57.891116 21141 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:57.897769 21264 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.901309 19842 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:57.910483 16403 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:57.922549 14342 catalog_manager.cc:5484] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:57.930442 16800 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:57.930470 14749 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.942909 16667 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.951676 14347 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.967121 16537 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.981087 14489 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:57.982813 21061 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.986379 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6c920958214148b3c3d5ac995e99e2 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:57.989427 16535 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:57.994365 14600 tablet_service.cc:1514] Processing DeleteTablet for tablet 55067e08fb694e90b6366d41cc962095 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:58.019002 21394 tablet_replica.cc:330] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:58.019380 21394 raft_consensus.cc:2229] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:58.023455 14344 catalog_manager.cc:5484] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:58.027643 21394 raft_consensus.cc:2258] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:58.028563 21394 ts_tablet_manager.cc:1875] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:58.035442 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:14:58.042737 14621 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:58.052255 21401 tablet_replica.cc:330] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:14:58.052587 21401 raft_consensus.cc:2229] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:58.052850 21401 raft_consensus.cc:2258] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:58.053637 21401 ts_tablet_manager.cc:1875] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:58.065412 21394 ts_tablet_manager.cc:1888] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:58.073096 21394 log.cc:1196] T 1d6c920958214148b3c3d5ac995e99e2 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d6c920958214148b3c3d5ac995e99e2
I20240503 08:14:58.075421 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 1d6c920958214148b3c3d5ac995e99e2 (table rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]) successfully deleted
I20240503 08:14:58.077569 21394 tablet_replica.cc:330] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:58.077996 21394 raft_consensus.cc:2229] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:58.078660 21394 raft_consensus.cc:2258] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:58.081723 21394 ts_tablet_manager.cc:1875] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:58.093955 21401 ts_tablet_manager.cc:1888] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:14:58.094467 21401 log.cc:1196] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/7cc0fe7d81e149d5b82fabfdc11d66b4
I20240503 08:14:58.095360 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 (table rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]) successfully deleted
I20240503 08:14:58.118811 21394 ts_tablet_manager.cc:1888] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:58.118970 21394 log.cc:1196] T 55067e08fb694e90b6366d41cc962095 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/55067e08fb694e90b6366d41cc962095
I20240503 08:14:58.119714 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 55067e08fb694e90b6366d41cc962095 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:14:58.188253 14878 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:58.191466 16800 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:58.191983 14489 consensus_queue.cc:237] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:58.192096 21264 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:58.195676 16537 raft_consensus.cc:1261] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:58.196940 21061 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:58.204166 21061 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:58.209798 16537 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:58.212643 14347 catalog_manager.cc:5484] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:14:58.220402 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa711e8c8564bc78846b16d68470cea with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:58.220952 21394 tablet_replica.cc:330] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:58.221349 21394 raft_consensus.cc:2229] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:58.221602 21394 raft_consensus.cc:2258] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:58.222333 21394 ts_tablet_manager.cc:1875] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:58.222612 14489 consensus_queue.cc:237] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:58.224884 16537 raft_consensus.cc:1261] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:58.225615 21061 consensus_queue.cc:1035] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:14:58.225991 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa711e8c8564bc78846b16d68470cea. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.228343 21264 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:58.229434 19842 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:58.233229 16536 raft_consensus.cc:2932] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:58.242835 21394 ts_tablet_manager.cc:1888] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:58.242969 21394 log.cc:1196] T dfa711e8c8564bc78846b16d68470cea P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/dfa711e8c8564bc78846b16d68470cea
I20240503 08:14:58.245990 16800 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:58.247011 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet dfa711e8c8564bc78846b16d68470cea (table rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]) successfully deleted
I20240503 08:14:58.248135 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dfa711e8c8564bc78846b16d68470cea with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:14:58.249336 14347 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:14:58.251155 14344 catalog_manager.cc:5484] T dfa711e8c8564bc78846b16d68470cea P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:58.260095 14878 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
W20240503 08:14:58.263778 14815 consensus_peers.cc:485] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88404b9cdf574cb0bca1bb49348fb4cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.264346 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 88404b9cdf574cb0bca1bb49348fb4cd with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:14:58.264765 21394 tablet_replica.cc:330] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:14:58.265024 21394 raft_consensus.cc:2229] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:58.265260 21394 raft_consensus.cc:2258] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:58.271122 16800 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:58.272255 21380 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:58.274966 21380 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:58.275408 16800 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:58.280869 14344 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:14:58.283092 14334 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 88404b9cdf574cb0bca1bb49348fb4cd with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:14:58.309474 21394 ts_tablet_manager.cc:1875] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:58.332247 21394 ts_tablet_manager.cc:1888] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:58.332473 21394 log.cc:1196] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/88404b9cdf574cb0bca1bb49348fb4cd
I20240503 08:14:58.333221 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 88404b9cdf574cb0bca1bb49348fb4cd (table rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]) successfully deleted
I20240503 08:14:58.541484 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.553983 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.570797 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:58.622676 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:58.640532 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:58.685850 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.691510 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:58.749770 21578 ts_tablet_manager.cc:902] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:14:58.836463 21581 ts_tablet_manager.cc:902] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:14:58.843479 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.884233 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.920455 21578 tablet_copy_client.cc:308] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:14:58.921530 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 88404b9cdf574cb0bca1bb49348fb4cd from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:60169)
I20240503 08:14:58.921708 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 88404b9cdf574cb0bca1bb49348fb4cd from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:60169: session id = 0f8702fad8ea4d22840dab11bb27f7f2-88404b9cdf574cb0bca1bb49348fb4cd
I20240503 08:14:58.925208 14901 tablet_copy_source_session.cc:205] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:58.956446 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88404b9cdf574cb0bca1bb49348fb4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.972199 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.982733 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.025993 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:59.032837 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.035578 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.076396 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:59.082016 21581 tablet_copy_client.cc:308] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:14:59.084615 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.086398 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet dfa711e8c8564bc78846b16d68470cea from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:14:59.086596 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet dfa711e8c8564bc78846b16d68470cea from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-dfa711e8c8564bc78846b16d68470cea
I20240503 08:14:59.088686 21578 tablet_copy_client.cc:783] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.088943 21578 tablet_copy_client.cc:647] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.091460 14507 tablet_copy_source_session.cc:205] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.123513 21581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa711e8c8564bc78846b16d68470cea. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:14:59.126384 14423 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 72800.2us
I20240503 08:14:59.156126 21578 tablet_copy_client.cc:523] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.172013 21578 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:14:59.176174 21581 tablet_copy_client.cc:783] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.176649 21581 tablet_copy_client.cc:647] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.191213 21581 tablet_copy_client.cc:523] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.195151 21581 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:14:59.486022 21581 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.490886 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.498528 21581 tablet_bootstrap.cc:492] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:14:59.500578 14489 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40736
I20240503 08:14:59.500815 14489 raft_consensus.cc:595] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:59.514221 21581 ts_tablet_manager.cc:1367] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.308s	user 0.052s	sys 0.008s
I20240503 08:14:59.517436 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.521616 21581 raft_consensus.cc:348] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:59.523104 21581 raft_consensus.cc:729] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.524319 21581 consensus_queue.cc:260] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:59.534878 21581 ts_tablet_manager.cc:1398] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240503 08:14:59.535233 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.563900 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.585183 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.593276 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-dfa711e8c8564bc78846b16d68470cea received from {username='slave'} at 127.2.49.135:44583
I20240503 08:14:59.593413 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-dfa711e8c8564bc78846b16d68470cea on tablet dfa711e8c8564bc78846b16d68470cea with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:59.595156 14490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40728
I20240503 08:14:59.595343 14490 raft_consensus.cc:595] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:59.598480 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.601766 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.625087 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.634475 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:59.638615 14468 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:14:59.642659 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.676584 21622 raft_consensus.cc:979] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: : Instructing follower 0f8702fad8ea4d22840dab11bb27f7f2 to start an election
I20240503 08:14:59.676714 21622 raft_consensus.cc:1067] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Signalling peer 0f8702fad8ea4d22840dab11bb27f7f2 to start an election
I20240503 08:14:59.680135 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:59.682861 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.691977 16403 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c"
dest_uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
 from {username='slave'} at 127.2.49.129:59005
I20240503 08:14:59.692333 16403 raft_consensus.cc:482] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:59.692546 16403 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.696378 16403 raft_consensus.cc:504] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:59.699584 16403 leader_election.cc:288] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [CANDIDATE]: Term 2 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:14:59.699872 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:14:59.700068 14621 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.703897 14621 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f8702fad8ea4d22840dab11bb27f7f2 in term 2.
I20240503 08:14:59.704674 16338 leader_election.cc:302] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 0f8702fad8ea4d22840dab11bb27f7f2; no voters: 
I20240503 08:14:59.705123 21393 raft_consensus.cc:979] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: : Instructing follower 0d77c4b1f12049dea71b97e86be2b288 to start an election
I20240503 08:14:59.705240 21393 raft_consensus.cc:1067] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Signalling peer 0d77c4b1f12049dea71b97e86be2b288 to start an election
I20240503 08:14:59.706636 16385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.709626 14621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71"
dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
 from {username='slave'} at 127.2.49.129:60669
I20240503 08:14:59.709914 14621 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:59.710054 14621 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.713207 14621 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:59.715692 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:59.715878 14490 raft_consensus.cc:3032] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:59.715962 14490 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:59.716158 14490 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:59.716477 14490 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.719766 14490 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f8702fad8ea4d22840dab11bb27f7f2 in term 2.
I20240503 08:14:59.729249 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.738677 16667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6835ad958e094ce4927135b942a2e8a8"
I20240503 08:14:59.738971 16667 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.741586 21578 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.741987 16667 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:14:59.742092 21578 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:14:59.742419 14621 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 2 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:14:59.743680 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:14:59.743918 14749 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.747035 14749 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:14:59.747968 14557 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 6835ad958e094ce4927135b942a2e8a8, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:14:59.748298 21697 raft_consensus.cc:2781] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:59.748411 21697 raft_consensus.cc:686] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:14:59.748725 21697 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:59.750932 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:14:59.751142 14490 raft_consensus.cc:3032] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:59.751236 14490 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:14:59.751474 14490 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:14:59.752022 14490 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:59.755602 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.757218 14490 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:14:59.757866 21730 raft_consensus.cc:2781] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:59.762781 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:14:59.777333 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:14:59.778466 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.792234 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.759717 21578 ts_tablet_manager.cc:1367] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.588s	user 0.059s	sys 0.003s
W20240503 08:14:59.761154 16338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 56592.1us
I20240503 08:14:59.805094 21578 raft_consensus.cc:348] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:59.805487 21578 raft_consensus.cc:729] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:14:59.805795 21578 consensus_queue.cc:260] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:59.817991 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:59.837975 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:14:59.877496 14555 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 107764us
I20240503 08:14:59.879387 21730 raft_consensus.cc:686] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 LEADER]: Becoming Leader. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Running, Role: LEADER
I20240503 08:14:59.884555 14344 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 1 to 2, leader changed from 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:14:59.886420 21730 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:59.892832 21578 ts_tablet_manager.cc:1398] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.090s	user 0.005s	sys 0.000s
I20240503 08:14:59.895234 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-88404b9cdf574cb0bca1bb49348fb4cd received from {username='slave'} at 127.2.49.133:60169
I20240503 08:14:59.895378 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-88404b9cdf574cb0bca1bb49348fb4cd on tablet 88404b9cdf574cb0bca1bb49348fb4cd with peer 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:59.953594 14687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:36937, user_credentials={real_user=slave}} blocked reactor thread for 44645.7us
I20240503 08:14:59.963356 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:14:59.967793 14349 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: term changed from 1 to 2, leader changed from 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) to 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133). New cstate: current_term: 2 leader_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:00.005579 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:00.025990 16667 raft_consensus.cc:1203] T dfa711e8c8564bc78846b16d68470cea P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:00.172371 16403 raft_consensus.cc:1203] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:00.203300 16667 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:15:00.204145 21697 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:00.248719 14749 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:15:00.249457 21736 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:00.252257 14489 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40806
I20240503 08:15:00.252452 14489 raft_consensus.cc:595] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 08:15:00.252521 14489 raft_consensus.cc:603] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 08:15:00.283841 14489 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:15:00.285233 21736 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:00.397787 14489 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:00.400816 16403 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:00.401660 21622 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:00.413882 21622 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:00.417851 16404 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:00.424999 14489 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:00.425937 21730 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:00.443055 14342 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:00.465436 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet a87c5af03f4d47d38bbfbd225c7195d1 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:00.477365 14490 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:15:00.480597 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87c5af03f4d47d38bbfbd225c7195d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:00.486328 14621 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:00.508355 16403 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:00.520943 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:00.525259 16667 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:00.525830 21852 tablet_replica.cc:330] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:00.526087 21736 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:00.529590 14749 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:00.530871 21736 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.533977 16667 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.526238 21852 raft_consensus.cc:2229] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:00.538028 21852 raft_consensus.cc:2258] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:00.538849 21852 ts_tablet_manager.cc:1875] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:00.539373 21249 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:00.557328 14342 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.565305 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a5221747794a13aa4ed763006ba291 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:00.577761 21866 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:00.578784 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:00.589387 14749 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.595197 21852 ts_tablet_manager.cc:1888] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:00.595455 21852 log.cc:1196] T a87c5af03f4d47d38bbfbd225c7195d1 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/a87c5af03f4d47d38bbfbd225c7195d1
I20240503 08:15:00.596238 21865 tablet_replica.cc:330] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:00.596522 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet a87c5af03f4d47d38bbfbd225c7195d1 (table rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]) successfully deleted
I20240503 08:15:00.597859 21865 raft_consensus.cc:2229] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:00.613407 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:00.657943 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:00.670039 14424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 83425.7us
W20240503 08:15:00.670039 21862 log.cc:924] Time spent T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20240503 08:15:00.671473 21865 raft_consensus.cc:2258] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:00.672222 21865 ts_tablet_manager.cc:1875] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:00.673429 14621 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:00.674525 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:00.679535 21865 ts_tablet_manager.cc:1888] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:15:00.679651 21865 log.cc:1196] T f7a5221747794a13aa4ed763006ba291 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7a5221747794a13aa4ed763006ba291
I20240503 08:15:00.680418 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet f7a5221747794a13aa4ed763006ba291 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:00.681325 21866 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.686779 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a87c5af03f4d47d38bbfbd225c7195d1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:00.680732 21730 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240503 08:15:00.689642 14349 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.707559 16403 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:00.762712 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:00.772989 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:00.775192 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:00.776367 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:00.851342 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:00.854967 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:00.869184 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:00.869212 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:00.892593 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:00.922751 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:00.948734 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:00.958022 14687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:36937, user_credentials={real_user=slave}} blocked reactor thread for 70694.1us
W20240503 08:15:00.969473 14817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:36937, user_credentials={real_user=slave}} blocked reactor thread for 59073.5us
I20240503 08:15:01.005647 21912 ts_tablet_manager.cc:902] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:01.049427 21912 tablet_copy_client.cc:308] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:01.050318 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet a87c5af03f4d47d38bbfbd225c7195d1 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:15:01.050477 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet a87c5af03f4d47d38bbfbd225c7195d1 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-a87c5af03f4d47d38bbfbd225c7195d1
I20240503 08:15:01.052425 14507 tablet_copy_source_session.cc:205] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:01.054039 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c5af03f4d47d38bbfbd225c7195d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:01.104880 21912 tablet_copy_client.cc:783] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:01.105211 21912 tablet_copy_client.cc:647] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:01.107744 21912 tablet_copy_client.cc:523] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:01.125103 21912 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:15:01.246662 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:01.256829 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:01.284301 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:01.300457 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:01.316629 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:01.321136 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:01.354585 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:01.419276 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:01.491616 21912 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:01.493402 21912 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:15:01.508049 21912 ts_tablet_manager.cc:1367] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.383s	user 0.043s	sys 0.013s
I20240503 08:15:01.509236 21912 raft_consensus.cc:348] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:01.510617 21912 raft_consensus.cc:729] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:15:01.511075 21912 consensus_queue.cc:260] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:01.539288 21912 ts_tablet_manager.cc:1398] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.031s	user 0.003s	sys 0.001s
I20240503 08:15:01.541900 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-a87c5af03f4d47d38bbfbd225c7195d1 received from {username='slave'} at 127.2.49.135:44583
I20240503 08:15:01.542060 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-a87c5af03f4d47d38bbfbd225c7195d1 on tablet a87c5af03f4d47d38bbfbd225c7195d1 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:15:01.545512 16667 raft_consensus.cc:1203] T a87c5af03f4d47d38bbfbd225c7195d1 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 08:15:01.601595 14424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 51430.9us
I20240503 08:15:01.760911 14621 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:01.781037 16667 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:01.783000 21978 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:01.801648 14749 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:01.802728 21978 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:01.817404 21736 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:01.822242 14749 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:01.829191 14349 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
W20240503 08:15:01.860323 14817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.129:37829, user_credentials={real_user=slave}} blocked reactor thread for 60559.3us
I20240503 08:15:01.866892 16669 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:01.930444 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:01.948884 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:01.974346 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:02.005923 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:02.030277 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:02.069252 22098 tablet_replica.cc:330] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:02.069574 22098 raft_consensus.cc:2229] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:02.070776 22098 raft_consensus.cc:2258] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:02.071883 22098 ts_tablet_manager.cc:1875] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:02.117897 16403 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:02.136072 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.139535 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.154281 14621 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:02.155066 21860 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:02.165143 22098 ts_tablet_manager.cc:1888] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:15:02.165278 22098 log.cc:1196] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:02.166216 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 4e86f4f3a01048e69f553c6557d79b71 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:02.173348 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:02.178062 22114 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:02.184628 14621 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:02.185577 14347 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 5 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:02.202613 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet f65ab6dba1124573962e2da58aa1859c with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:02.203140 22098 tablet_replica.cc:330] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:02.203352 22098 raft_consensus.cc:2229] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:02.203517 22098 raft_consensus.cc:2258] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:02.204281 22098 ts_tablet_manager.cc:1875] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:02.223994 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:02.228358 22098 ts_tablet_manager.cc:1888] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:15:02.228581 22098 log.cc:1196] T f65ab6dba1124573962e2da58aa1859c P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/f65ab6dba1124573962e2da58aa1859c
I20240503 08:15:02.235412 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet f65ab6dba1124573962e2da58aa1859c (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:15:02.259418 16403 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:02.290942 14621 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:02.293759 21860 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:02.297382 22114 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:02.302239 14621 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:02.327102 14347 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 2 leader_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:02.340034 14334 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f65ab6dba1124573962e2da58aa1859c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 08:15:02.430207 16340 consensus_peers.cc:485] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f65ab6dba1124573962e2da58aa1859c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:02.601728 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.629948 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:02.644845 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:02.706928 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.719014 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:02.732527 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:02.740589 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.766098 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:02.773968 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.782626 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:02.795068 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:02.861943 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.870646 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:02.880897 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:02.913180 22227 ts_tablet_manager.cc:902] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:15:02.931243 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:02.947655 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:02.959671 22227 tablet_copy_client.cc:308] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.133:38023
I20240503 08:15:03.010337 14749 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:03.052498 16800 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:03.054942 21914 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:03.077104 16425 tablet_copy_service.cc:140] P 0f8702fad8ea4d22840dab11bb27f7f2: Received BeginTabletCopySession request for tablet f65ab6dba1124573962e2da58aa1859c from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:39289)
I20240503 08:15:03.077353 16425 tablet_copy_service.cc:161] P 0f8702fad8ea4d22840dab11bb27f7f2: Beginning new tablet copy session on tablet f65ab6dba1124573962e2da58aa1859c from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:39289: session id = 6835ad958e094ce4927135b942a2e8a8-f65ab6dba1124573962e2da58aa1859c
I20240503 08:15:03.081449 16425 tablet_copy_source_session.cc:205] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:03.083420 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65ab6dba1124573962e2da58aa1859c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.090785 22227 tablet_copy_client.cc:783] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:03.091565 22227 tablet_copy_client.cc:647] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:03.099962 21911 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:03.100793 16801 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:03.109185 14342 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 5 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:03.111244 22227 tablet_copy_client.cc:523] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:03.116758 22227 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:15:03.219806 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 897a701b880641b08816f2d1a254ef03 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:03.224417 14749 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
W20240503 08:15:03.228073 14687 consensus_peers.cc:485] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897a701b880641b08816f2d1a254ef03. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:03.229912 16801 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:03.231282 14469 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:03.235502 21911 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:03.242005 21911 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.245952 16801 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.263485 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:03.266172 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 897a701b880641b08816f2d1a254ef03 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:03.266909 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:03.268869 14344 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.281200 22227 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:03.281811 22227 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:15:03.282298 22227 ts_tablet_manager.cc:1367] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.166s	user 0.062s	sys 0.000s
I20240503 08:15:03.283155 22227 raft_consensus.cc:348] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:03.283505 22227 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:15:03.283825 22227 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:03.297472 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:03.303829 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:03.305588 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:03.352325 22227 ts_tablet_manager.cc:1398] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.070s	user 0.003s	sys 0.008s
I20240503 08:15:03.353220 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:03.353335 16425 tablet_copy_service.cc:342] P 0f8702fad8ea4d22840dab11bb27f7f2: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-f65ab6dba1124573962e2da58aa1859c received from {username='slave'} at 127.2.49.135:39289
I20240503 08:15:03.353513 16425 tablet_copy_service.cc:433] P 0f8702fad8ea4d22840dab11bb27f7f2: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-f65ab6dba1124573962e2da58aa1859c on tablet f65ab6dba1124573962e2da58aa1859c with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:15:03.364807 22280 tablet_replica.cc:330] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:03.365164 22280 raft_consensus.cc:2229] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:03.365692 22280 raft_consensus.cc:2258] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:03.366817 22280 ts_tablet_manager.cc:1875] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:03.420118 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:03.464751 22280 ts_tablet_manager.cc:1888] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:03.465888 22280 log.cc:1196] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/897a701b880641b08816f2d1a254ef03
I20240503 08:15:03.467216 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 897a701b880641b08816f2d1a254ef03 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:15:03.565706 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:03.584115 14621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31"
dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43458
I20240503 08:15:03.584401 14621 raft_consensus.cc:595] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:15:03.612926 14599 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:03.627388 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:03.722577 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:03.734467 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:03.745726 22333 ts_tablet_manager.cc:902] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:15:03.748836 22344 raft_consensus.cc:979] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: : Instructing follower 036223b0890d4634acfadf9596f39f80 to start an election
I20240503 08:15:03.749109 22344 raft_consensus.cc:1067] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Signalling peer 036223b0890d4634acfadf9596f39f80 to start an election
I20240503 08:15:03.758204 16801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31"
dest_uuid: "036223b0890d4634acfadf9596f39f80"
 from {username='slave'} at 127.2.49.130:39317
I20240503 08:15:03.758531 16801 raft_consensus.cc:482] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:03.758800 16801 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:03.762507 16801 raft_consensus.cc:504] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:03.765466 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "036223b0890d4634acfadf9596f39f80" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:15:03.765676 14621 raft_consensus.cc:3032] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:03.765663 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "036223b0890d4634acfadf9596f39f80" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:15:03.765753 14621 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:15:03.765856 14749 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:03.765894 14878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "036223b0890d4634acfadf9596f39f80" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
I20240503 08:15:03.766064 14878 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:03.766067 14621 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:03.766543 14621 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:03.768745 14878 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036223b0890d4634acfadf9596f39f80 in term 2.
I20240503 08:15:03.768745 14749 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036223b0890d4634acfadf9596f39f80 in term 2.
I20240503 08:15:03.769299 14621 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036223b0890d4634acfadf9596f39f80 in term 2.
I20240503 08:15:03.769354 16735 leader_election.cc:302] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 036223b0890d4634acfadf9596f39f80, b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:15:03.769779 16801 leader_election.cc:288] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [CANDIDATE]: Term 2 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:03.769805 22258 raft_consensus.cc:2781] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:03.774495 14489 consensus_queue.cc:237] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:03.783344 22258 raft_consensus.cc:686] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Becoming Leader. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Running, Role: LEADER
I20240503 08:15:03.783526 16536 raft_consensus.cc:1261] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:03.784055 22258 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:03.785751 22341 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:03.797930 14749 raft_consensus.cc:1261] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:03.798760 22341 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:03.801653 22265 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.807861 14342 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 reported cstate change: term changed from 1 to 2, leader changed from 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) to 036223b0890d4634acfadf9596f39f80 (127.2.49.136). New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:03.809723 16536 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.816908 14751 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.824967 14344 catalog_manager.cc:5484] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 4 to 5, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:03.834050 16667 raft_consensus.cc:1203] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:15:03.859061 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:03.869869 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:03.871585 22333 tablet_copy_client.cc:308] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:15:03.874780 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 897a701b880641b08816f2d1a254ef03 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:44645)
I20240503 08:15:03.874956 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 897a701b880641b08816f2d1a254ef03 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:44645: session id = 95860e22c9474e2d894f04deba8442cc-897a701b880641b08816f2d1a254ef03
I20240503 08:15:03.878764 14771 tablet_copy_source_session.cc:205] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:03.886330 22333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897a701b880641b08816f2d1a254ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:03.886425 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:03.929890 22333 tablet_copy_client.cc:783] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:03.932024 22333 tablet_copy_client.cc:647] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:03.934259 22333 tablet_copy_client.cc:523] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:15:03.936923 14425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 105124us
I20240503 08:15:03.947512 22333 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:15:04.047035 14878 consensus_queue.cc:237] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:04.052000 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c687520c0a349fd8438dacbd5011001 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48320
I20240503 08:15:04.061771 22397 tablet_replica.cc:330] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:15:04.061997 22397 raft_consensus.cc:2229] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:04.063910 16404 raft_consensus.cc:1261] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:04.064687 22161 consensus_queue.cc:1035] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:04.065150 22397 raft_consensus.cc:2258] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:04.066627 22397 ts_tablet_manager.cc:1875] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:04.067615 14749 raft_consensus.cc:1261] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:04.068229 22161 consensus_queue.cc:1035] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:04.068969 22333 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:04.069545 22333 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:15:04.069851 22333 ts_tablet_manager.cc:1367] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.122s	user 0.051s	sys 0.011s
I20240503 08:15:04.070822 22333 raft_consensus.cc:348] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:04.071146 22333 raft_consensus.cc:729] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:15:04.071461 22333 consensus_queue.cc:260] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:04.072223 22333 ts_tablet_manager.cc:1398] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:15:04.077594 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-897a701b880641b08816f2d1a254ef03 received from {username='slave'} at 127.2.49.134:44645
I20240503 08:15:04.077734 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-897a701b880641b08816f2d1a254ef03 on tablet 897a701b880641b08816f2d1a254ef03 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:15:04.083066 22397 ts_tablet_manager.cc:1888] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:15:04.083344 22397 log.cc:1196] T 1c687520c0a349fd8438dacbd5011001 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c687520c0a349fd8438dacbd5011001
I20240503 08:15:04.084280 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 1c687520c0a349fd8438dacbd5011001 (table rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]) successfully deleted
I20240503 08:15:04.120898 16536 raft_consensus.cc:1203] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:04.123963 14749 raft_consensus.cc:2932] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:04.122478 22412 raft_consensus.cc:2932] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:04.132246 14349 catalog_manager.cc:5484] T 12df78f752c34fe3885b235c4a19e325 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:04.135941 16404 raft_consensus.cc:2932] T 12df78f752c34fe3885b235c4a19e325 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:04.139073 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 12df78f752c34fe3885b235c4a19e325 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:04.161301 22416 tablet_replica.cc:330] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:04.163323 22416 raft_consensus.cc:2229] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:04.163719 22416 raft_consensus.cc:2258] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:04.164853 22416 ts_tablet_manager.cc:1875] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:04.165493 14621 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:15:04.166462 22258 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:04.180191 22416 ts_tablet_manager.cc:1888] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:15:04.181931 22416 log.cc:1196] T 12df78f752c34fe3885b235c4a19e325 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/12df78f752c34fe3885b235c4a19e325
I20240503 08:15:04.183302 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 12df78f752c34fe3885b235c4a19e325 (table rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]) successfully deleted
I20240503 08:15:04.234930 14749 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:15:04.235677 22258 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:04.241204 14878 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:15:04.242013 22258 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:04.354126 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:04.433822 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:04.470834 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:04.490795 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:04.536928 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:04.561319 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:04.565867 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:04.661010 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:04.686682 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:04.732161 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:04.736228 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:04.763087 14878 consensus_queue.cc:237] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:04.769265 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:04.824465 16801 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:04.825172 22412 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:04.827832 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:04.846062 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:04.851222 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:04.879616 14621 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:04.880223 22412 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:15:04.917110 14557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.134:36937, user_credentials={real_user=slave}} blocked reactor thread for 76273.6us
I20240503 08:15:04.935518 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:04.937000 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:04.937885 22539 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:04.939702 16800 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:04.948415 14347 catalog_manager.cc:5484] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:04.985512 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:04.987079 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:04.995131 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:04.997779 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:05.007514 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 0780fb979eee4bae98a014d9721c9441 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:05.014724 14619 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:05.052340 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:05.093086 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:05.098766 22547 tablet_replica.cc:330] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:05.099098 22547 raft_consensus.cc:2229] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:05.099812 22547 raft_consensus.cc:2258] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:05.107017 22547 ts_tablet_manager.cc:1875] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:05.112967 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:05.156668 22547 ts_tablet_manager.cc:1888] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:15:05.169885 22547 log.cc:1196] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/0780fb979eee4bae98a014d9721c9441
I20240503 08:15:05.171006 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 0780fb979eee4bae98a014d9721c9441 (table rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]) successfully deleted
I20240503 08:15:05.383780 14621 consensus_queue.cc:237] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:05.395730 16667 raft_consensus.cc:1261] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:05.396865 22330 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:05.442059 22330 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:05.446465 16669 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:05.451969 14347 catalog_manager.cc:5484] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:05.459105 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a216295a8a4feaa2d3326035879073 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48320
I20240503 08:15:05.463869 14621 consensus_queue.cc:237] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
W20240503 08:15:05.469503 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76a216295a8a4feaa2d3326035879073. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:05.476107 22611 tablet_replica.cc:330] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:15:05.476958 22611 raft_consensus.cc:2229] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:05.477357 22611 raft_consensus.cc:2258] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:05.478327 22611 ts_tablet_manager.cc:1875] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:05.483461 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:05.493165 22611 ts_tablet_manager.cc:1888] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:05.493333 22611 log.cc:1196] T 76a216295a8a4feaa2d3326035879073 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/76a216295a8a4feaa2d3326035879073
I20240503 08:15:05.501055 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 76a216295a8a4feaa2d3326035879073 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:15:05.510617 16667 raft_consensus.cc:1261] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:05.523144 22538 consensus_queue.cc:1035] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:05.557967 22538 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:05.562425 14333 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 76a216295a8a4feaa2d3326035879073 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:05.587221 16667 raft_consensus.cc:2932] T 76a216295a8a4feaa2d3326035879073 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:05.600170 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:05.606420 14347 catalog_manager.cc:5484] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:05.611606 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:05.612115 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:05.640527 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:05.643177 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:05.667905 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:05.672224 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:05.982136 14621 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:05.995272 16404 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:05.995987 22538 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:06.012081 22690 ts_tablet_manager.cc:902] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:15:06.013221 16800 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:06.022944 14489 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:06.025177 22538 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:06.029146 22538 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:06.033059 14489 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:06.051584 22690 tablet_copy_client.cc:308] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:15:06.047753 14349 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:06.052330 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 76a216295a8a4feaa2d3326035879073 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:48741)
I20240503 08:15:06.052479 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 76a216295a8a4feaa2d3326035879073 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:48741: session id = 95860e22c9474e2d894f04deba8442cc-76a216295a8a4feaa2d3326035879073
I20240503 08:15:06.056131 14641 tablet_copy_source_session.cc:205] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:06.057798 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet dd11460051ab40febc2fe6aff58b5028 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:06.075160 14878 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:06.075937 22258 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:06.083539 14749 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:06.084311 22258 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:06.088805 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a216295a8a4feaa2d3326035879073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:06.104955 22703 tablet_replica.cc:330] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:06.105383 22703 raft_consensus.cc:2229] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:06.105820 22703 raft_consensus.cc:2258] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:06.106858 22703 ts_tablet_manager.cc:1875] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:06.119786 14878 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.120417 16405 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:06.125902 14349 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 4 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.133381 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 169bd9ea6772420abf757dd55cf44d31 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:06.139458 22721 tablet_replica.cc:330] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:06.139842 22721 raft_consensus.cc:2229] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:06.140126 22721 raft_consensus.cc:2258] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:06.141002 22721 ts_tablet_manager.cc:1875] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:06.141119 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:06.154946 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:06.156169 16737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:45663, user_credentials={real_user=slave}} blocked reactor thread for 45248us
I20240503 08:15:06.164636 22690 tablet_copy_client.cc:783] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:06.165053 22690 tablet_copy_client.cc:647] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:06.167997 22690 tablet_copy_client.cc:523] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:06.169718 22703 ts_tablet_manager.cc:1888] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:15:06.170089 22703 log.cc:1196] T dd11460051ab40febc2fe6aff58b5028 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd11460051ab40febc2fe6aff58b5028
I20240503 08:15:06.171590 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet dd11460051ab40febc2fe6aff58b5028 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:06.174697 22690 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:15:06.175920 22721 ts_tablet_manager.cc:1888] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:15:06.176442 22721 log.cc:1196] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:06.178097 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 169bd9ea6772420abf757dd55cf44d31 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:06.192229 14749 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
W20240503 08:15:06.212911 16735 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.132:35075, user_credentials={real_user=slave}} blocked reactor thread for 88927.2us
I20240503 08:15:06.110059 22714 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.220082 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:06.272393 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:06.307256 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:06.310698 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:06.311244 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:06.326004 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:06.410871 22690 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:06.414237 22690 tablet_bootstrap.cc:492] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:15:06.414850 22690 ts_tablet_manager.cc:1367] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.240s	user 0.056s	sys 0.009s
I20240503 08:15:06.450708 22690 raft_consensus.cc:348] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:06.450986 22690 raft_consensus.cc:729] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:15:06.457314 22690 consensus_queue.cc:260] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:06.468185 22690 ts_tablet_manager.cc:1398] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.053s	user 0.000s	sys 0.003s
I20240503 08:15:06.472257 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-76a216295a8a4feaa2d3326035879073 received from {username='slave'} at 127.2.49.134:48741
I20240503 08:15:06.472389 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-76a216295a8a4feaa2d3326035879073 on tablet 76a216295a8a4feaa2d3326035879073 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:15:06.489691 14749 consensus_queue.cc:237] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:06.519377 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:06.524137 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:06.563735 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:06.564977 16801 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:06.565709 21911 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:06.573444 16404 raft_consensus.cc:1261] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:06.574607 21911 consensus_queue.cc:1035] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:06.589390 16536 raft_consensus.cc:1203] T 76a216295a8a4feaa2d3326035879073 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:06.598914 21911 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.613231 16404 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.616581 14349 catalog_manager.cc:5484] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.617406 16800 raft_consensus.cc:2932] T 1d6c920958214148b3c3d5ac995e99e2 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:06.634974 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:06.639315 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6c920958214148b3c3d5ac995e99e2 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:06.665158 14469 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:06.687331 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:06.706813 22791 tablet_replica.cc:330] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:06.707129 22791 raft_consensus.cc:2229] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:06.707410 22791 raft_consensus.cc:2258] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:06.708410 22791 ts_tablet_manager.cc:1875] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:06.713745 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:06.725275 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:06.755352 22791 ts_tablet_manager.cc:1888] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:06.759192 22791 log.cc:1196] T 1d6c920958214148b3c3d5ac995e99e2 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d6c920958214148b3c3d5ac995e99e2
I20240503 08:15:06.760897 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 1d6c920958214148b3c3d5ac995e99e2 (table rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]) successfully deleted
I20240503 08:15:07.184929 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:07.204280 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:07.283751 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:07.310178 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:07.320850 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:07.331712 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:07.332738 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:07.347976 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:07.362140 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:07.398828 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:07.464953 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:07.468899 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:07.471316 14749 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:07.483099 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:07.527526 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:07.528316 16800 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:07.528874 22788 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:07.543504 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:07.544356 22788 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:07.553251 16800 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:07.563769 14349 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 5 to 6, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:07.601069 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:07.659157 14749 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:07.662135 16800 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:07.658082 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet c130eb7ea8a74f4b8d9b79e23af70067 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:07.666069 22793 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:15:07.667346 14685 consensus_peers.cc:485] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c130eb7ea8a74f4b8d9b79e23af70067. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:07.670132 22793 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:07.674314 16800 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:07.677752 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c130eb7ea8a74f4b8d9b79e23af70067 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:07.681952 14349 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:07.689132 22980 tablet_replica.cc:330] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:07.693459 22980 raft_consensus.cc:2229] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:07.693804 22980 raft_consensus.cc:2258] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:07.695061 22980 ts_tablet_manager.cc:1875] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:07.722810 22980 ts_tablet_manager.cc:1888] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:07.724251 22980 log.cc:1196] T c130eb7ea8a74f4b8d9b79e23af70067 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/c130eb7ea8a74f4b8d9b79e23af70067
I20240503 08:15:07.732187 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet c130eb7ea8a74f4b8d9b79e23af70067 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:07.893658 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:07.941679 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:07.991338 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:07.996760 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:08.084100 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:08.094245 14489 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:08.114321 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:08.117534 16667 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:08.118553 22535 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:08.124907 14878 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:15:08.125710 22535 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:15:08.136307 22535 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:08.138994 14880 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:08.151152 14349 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 4 to 5, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:08.156998 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:08.158691 23068 tablet_replica.cc:330] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:08.160187 23068 raft_consensus.cc:2229] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:08.161597 23068 raft_consensus.cc:2258] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:08.163872 23068 ts_tablet_manager.cc:1875] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:08.173681 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:08.214581 23060 ts_tablet_manager.cc:902] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:15:08.220943 16669 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:08.221976 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:08.229480 23068 ts_tablet_manager.cc:1888] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:15:08.229974 23068 log.cc:1196] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:08.235515 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:08.247578 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:08.265453 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:08.309187 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet da1f165b4a5647a48ad7ff951fb3fbf6 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:08.315255 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:08.331001 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:08.361188 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:08.367548 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:08.367870 14489 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:08.371539 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:08.406383 23060 tablet_copy_client.cc:308] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:15:08.407114 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet c130eb7ea8a74f4b8d9b79e23af70067 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:37499)
I20240503 08:15:08.407263 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet c130eb7ea8a74f4b8d9b79e23af70067 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:37499: session id = 0f8702fad8ea4d22840dab11bb27f7f2-c130eb7ea8a74f4b8d9b79e23af70067
I20240503 08:15:08.410478 14771 tablet_copy_source_session.cc:205] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:08.411959 23060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130eb7ea8a74f4b8d9b79e23af70067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:08.414809 16537 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:08.417079 22535 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:08.423326 16800 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:08.423975 22535 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:08.439595 22535 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:08.442981 16800 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:08.449009 14349 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:08.460318 16536 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:08.462821 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b677894b51f4dc58f157d8f4d632e9a with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:08.463171 23068 tablet_replica.cc:330] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:08.463356 23068 raft_consensus.cc:2229] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:08.463534 23068 raft_consensus.cc:2258] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:08.464241 23068 ts_tablet_manager.cc:1875] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:08.473690 23068 ts_tablet_manager.cc:1888] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:08.473912 23068 log.cc:1196] T 6b677894b51f4dc58f157d8f4d632e9a P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/6b677894b51f4dc58f157d8f4d632e9a
I20240503 08:15:08.475876 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 6b677894b51f4dc58f157d8f4d632e9a (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:15:08.494740 23060 tablet_copy_client.cc:783] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:08.495136 23060 tablet_copy_client.cc:647] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:08.497085 23060 tablet_copy_client.cc:523] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:08.501641 23060 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:15:08.613647 23060 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:08.614259 23060 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:15:08.614619 23060 ts_tablet_manager.cc:1367] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.113s	user 0.073s	sys 0.015s
I20240503 08:15:08.615542 23060 raft_consensus.cc:348] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:08.615823 23060 raft_consensus.cc:729] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:15:08.616173 23060 consensus_queue.cc:260] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:08.617738 23060 ts_tablet_manager.cc:1398] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:15:08.620111 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-c130eb7ea8a74f4b8d9b79e23af70067 received from {username='slave'} at 127.2.49.133:37499
I20240503 08:15:08.620265 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-c130eb7ea8a74f4b8d9b79e23af70067 on tablet c130eb7ea8a74f4b8d9b79e23af70067 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:15:08.740473 14621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946"
dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43696
I20240503 08:15:08.740677 14621 raft_consensus.cc:595] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:15:08.778291 14749 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:08.788503 16800 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:08.789417 22793 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:08.795442 16537 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:08.796139 22793 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:08.802522 22935 raft_consensus.cc:979] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: : Instructing follower 129a2ac87d7f416db6d47d79bb1c2b5f to start an election
I20240503 08:15:08.802652 22935 raft_consensus.cc:1067] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Signalling peer 129a2ac87d7f416db6d47d79bb1c2b5f to start an election
I20240503 08:15:08.803426 14489 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
 from {username='slave'} at 127.2.49.130:47845
I20240503 08:15:08.803719 14489 raft_consensus.cc:482] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:08.803855 14489 raft_consensus.cc:3037] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:08.809587 14489 raft_consensus.cc:504] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:08.811486 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:15:08.811784 14621 raft_consensus.cc:3032] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:08.811908 14621 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:15:08.812261 14621 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:08.812713 14621 raft_consensus.cc:3037] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:08.819368 14621 raft_consensus.cc:2454] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 2.
I20240503 08:15:08.820012 14423 leader_election.cc:302] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:15:08.820389 22535 raft_consensus.cc:2781] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:08.820539 22535 raft_consensus.cc:686] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:15:08.820848 22535 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:08.822867 14489 leader_election.cc:288] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 2 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937)
I20240503 08:15:08.823544 16536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c595bb19d6946b0b1f853064892e946" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "95860e22c9474e2d894f04deba8442cc"
I20240503 08:15:08.823880 16536 raft_consensus.cc:3037] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:08.824542 14347 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 1 to 2, leader changed from 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:08.827250 16536 raft_consensus.cc:2454] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 2.
I20240503 08:15:08.831849 23178 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:08.832706 16800 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:08.838201 14347 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:08.843860 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet f19b7ef418974b06bdcc7125e8cf732d with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48320
I20240503 08:15:08.852875 16535 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:08.878557 23210 tablet_replica.cc:330] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:15:08.878885 23210 raft_consensus.cc:2229] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:08.879176 23210 raft_consensus.cc:2258] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:08.879956 23210 ts_tablet_manager.cc:1875] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:08.944700 23210 ts_tablet_manager.cc:1888] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:08.944860 23210 log.cc:1196] T f19b7ef418974b06bdcc7125e8cf732d P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/f19b7ef418974b06bdcc7125e8cf732d
I20240503 08:15:08.945797 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet f19b7ef418974b06bdcc7125e8cf732d (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:08.968515 16404 raft_consensus.cc:1203] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:09.322882 14621 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:09.324775 22535 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:09.381462 16537 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:09.385152 22535 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:09.441784 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:09.442896 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:09.502900 14468 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:15:09.510694 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:09.539214 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:09.545269 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:09.554652 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:09.557857 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:09.718407 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:09.721977 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:09.751660 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:09.763682 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:09.778698 14468 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:15:09.817685 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:09.822793 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:09.833143 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:10.295336 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:10.336405 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:10.356500 14489 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:10.358551 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:10.359139 16537 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:10.359944 22535 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:10.363685 22535 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:10.366952 16537 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:10.368521 14468 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:15:10.380584 14349 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:10.477680 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:10.478606 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:10.486107 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:10.527148 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:10.541533 14489 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:10.542320 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c595bb19d6946b0b1f853064892e946 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:10.549492 16537 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:10.550200 22370 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:15:10.552692 14425 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c595bb19d6946b0b1f853064892e946. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:10.555600 22370 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:10.558688 16537 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:10.568588 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:10.571702 23594 tablet_replica.cc:330] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:10.572021 23594 raft_consensus.cc:2229] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:10.572314 23594 raft_consensus.cc:2258] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:10.573169 23594 ts_tablet_manager.cc:1875] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:10.577744 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:10.582415 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:10.585352 23594 ts_tablet_manager.cc:1888] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:15:10.589814 23594 log.cc:1196] T 5c595bb19d6946b0b1f853064892e946 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c595bb19d6946b0b1f853064892e946
I20240503 08:15:10.593128 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 5c595bb19d6946b0b1f853064892e946 (table rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]) successfully deleted
I20240503 08:15:10.602209 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5c595bb19d6946b0b1f853064892e946 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:10.623113 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:10.624538 14341 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 7 to 8, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:10.637490 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:10.645534 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:10.688494 14468 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:15:10.696326 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:11.067571 23665 ts_tablet_manager.cc:902] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:11.150955 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:11.172551 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:11.179268 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:11.182446 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:11.206274 14880 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:11.211951 16537 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:11.212700 22936 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.213943 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:11.215708 16667 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:11.216645 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:11.216542 22936 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.245182 14489 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57570
I20240503 08:15:11.245523 14489 raft_consensus.cc:595] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Received request to transfer leadership
I20240503 08:15:11.262729 14468 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:15:11.270552 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:11.295241 22936 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:11.298352 16669 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:11.310788 14349 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:11.328302 16537 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:11.350327 22370 raft_consensus.cc:979] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: : Instructing follower 0d77c4b1f12049dea71b97e86be2b288 to start an election
I20240503 08:15:11.350483 22370 raft_consensus.cc:1067] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Signalling peer 0d77c4b1f12049dea71b97e86be2b288 to start an election
I20240503 08:15:11.351886 14619 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae"
dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
 from {username='slave'} at 127.2.49.129:60669
I20240503 08:15:11.352113 14619 raft_consensus.cc:482] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:11.352430 14619 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:11.355966 14619 raft_consensus.cc:504] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:11.357882 16535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "95860e22c9474e2d894f04deba8442cc"
I20240503 08:15:11.358129 16535 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:11.360956 16535 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:15:11.364375 16801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "036223b0890d4634acfadf9596f39f80"
I20240503 08:15:11.364686 16801 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:11.365260 14619 leader_election.cc:288] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 2 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937), 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:15:11.368252 16801 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:15:11.369145 14556 leader_election.cc:302] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 036223b0890d4634acfadf9596f39f80, 0d77c4b1f12049dea71b97e86be2b288, 95860e22c9474e2d894f04deba8442cc; no voters: 
I20240503 08:15:11.369614 23108 raft_consensus.cc:2781] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:11.369781 23108 raft_consensus.cc:686] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:15:11.370121 23108 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:11.375850 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea213e5a12452f8b894b9cdb94f9ae" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:15:11.376092 14490 raft_consensus.cc:3032] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:11.376155 14490 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:15:11.376444 14490 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:11.376854 14490 raft_consensus.cc:3037] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:11.380518 14490 raft_consensus.cc:2454] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:15:11.389707 23665 tablet_copy_client.cc:308] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:11.390625 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 5c595bb19d6946b0b1f853064892e946 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:55303)
I20240503 08:15:11.390799 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 5c595bb19d6946b0b1f853064892e946 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:55303: session id = 036223b0890d4634acfadf9596f39f80-5c595bb19d6946b0b1f853064892e946
I20240503 08:15:11.390791 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 55067e08fb694e90b6366d41cc962095 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:11.392057 23722 tablet_replica.cc:330] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:11.392731 23722 raft_consensus.cc:2229] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:11.393296 23722 raft_consensus.cc:2258] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:11.394394 23722 ts_tablet_manager.cc:1875] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:11.395627 14507 tablet_copy_source_session.cc:205] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:11.397166 23665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c595bb19d6946b0b1f853064892e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:11.399638 14349 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 1 to 2, leader changed from 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:11.404767 23665 tablet_copy_client.cc:783] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:11.405125 23665 tablet_copy_client.cc:647] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:11.410024 23722 ts_tablet_manager.cc:1888] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:11.410250 23722 log.cc:1196] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/55067e08fb694e90b6366d41cc962095
I20240503 08:15:11.411202 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 55067e08fb694e90b6366d41cc962095 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:11.412684 23665 tablet_copy_client.cc:523] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:11.418486 23665 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:15:11.498889 23665 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:11.533516 23665 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:15:11.533875 23665 ts_tablet_manager.cc:1367] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.115s	user 0.066s	sys 0.015s
I20240503 08:15:11.534704 23665 raft_consensus.cc:348] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:11.534919 23665 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:15:11.535131 23665 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:11.537403 23665 ts_tablet_manager.cc:1398] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:15:11.538367 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-5c595bb19d6946b0b1f853064892e946 received from {username='slave'} at 127.2.49.136:55303
I20240503 08:15:11.538501 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 036223b0890d4634acfadf9596f39f80-5c595bb19d6946b0b1f853064892e946 on tablet 5c595bb19d6946b0b1f853064892e946 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:15:11.778648 14880 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:11.783492 16801 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:11.784334 23108 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.789515 16537 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:11.790779 16404 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:11.791857 22936 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.793722 23603 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.862689 23603 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:11.880461 16535 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:11.882723 23108 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.886873 14349 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:11.893373 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d16c2981213464cbefa44b6ca509182 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:11.910980 16404 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:11.954777 14490 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:11.955750 23108 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:11.959414 16537 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:11.969251 23833 tablet_replica.cc:330] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:11.969866 23833 raft_consensus.cc:2229] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:11.970715 23833 raft_consensus.cc:2258] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:11.977118 23833 ts_tablet_manager.cc:1875] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:11.994625 23833 ts_tablet_manager.cc:1888] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:11.995311 23833 log.cc:1196] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/5d16c2981213464cbefa44b6ca509182
I20240503 08:15:12.008297 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 5d16c2981213464cbefa44b6ca509182 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:15:12.070976 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:12.089565 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:12.116278 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:12.117487 16801 raft_consensus.cc:1203] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:15:12.132753 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:12.178100 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:12.183320 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:12.239830 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:12.244084 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:12.281527 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:12.292681 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:12.301460 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:12.311862 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:12.399392 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:12.429427 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:12.470274 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:12.480406 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:12.789521 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:12.789862 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:12.808384 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:12.814473 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:12.840380 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:12.851086 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:12.919694 14860 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:12.972127 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:13.033756 14490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43830
I20240503 08:15:13.033977 14490 raft_consensus.cc:595] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Received request to transfer leadership
I20240503 08:15:13.100286 23993 raft_consensus.cc:979] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: : Instructing follower b35c4c5d528c4a8f9e754ff9481c3938 to start an election
I20240503 08:15:13.100442 23993 raft_consensus.cc:1067] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Signalling peer b35c4c5d528c4a8f9e754ff9481c3938 to start an election
I20240503 08:15:13.101631 14880 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9"
dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
 from {username='slave'} at 127.2.49.129:53921
I20240503 08:15:13.102011 14880 raft_consensus.cc:482] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:13.102134 14880 raft_consensus.cc:3037] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.106702 14880 raft_consensus.cc:504] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:13.117641 14880 leader_election.cc:288] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 2 election: Requested vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:15:13.119109 14490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:15:13.119338 14490 raft_consensus.cc:3032] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:13.119402 14490 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:15:13.119654 14490 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:13.120013 14490 raft_consensus.cc:3037] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.124588 16405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a949e76ab5477680a4f9912e9672e9" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
I20240503 08:15:13.124922 16405 raft_consensus.cc:3037] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:13.129428 16405 raft_consensus.cc:2454] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 2.
I20240503 08:15:13.129994 14815 leader_election.cc:302] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f8702fad8ea4d22840dab11bb27f7f2, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:15:13.130370 22936 raft_consensus.cc:2781] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:13.139614 22936 raft_consensus.cc:686] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:15:13.140077 22936 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:13.150264 14342 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 1 to 2, leader changed from 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:13.177915 14490 raft_consensus.cc:2454] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 2.
I20240503 08:15:13.287935 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:13.310575 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:13.347352 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:13.353536 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:15:13.400924 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 50878.5us
I20240503 08:15:13.404330 14619 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:13.412817 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:13.423766 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:13.433708 16801 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:13.434367 23108 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:13.441289 16535 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:13.441953 23108 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:13.468784 24073 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.469592 16535 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.478111 16800 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.484326 14342 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 6 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.492556 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fea213e5a12452f8b894b9cdb94f9ae with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:13.501144 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:13.548172 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:13.575155 24079 tablet_replica.cc:330] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:13.575377 24079 raft_consensus.cc:2229] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:13.575552 24079 raft_consensus.cc:2258] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:13.576622 24079 ts_tablet_manager.cc:1875] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:13.603801 14490 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:13.606984 22936 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:13.685585 24079 ts_tablet_manager.cc:1888] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:15:13.685716 24079 log.cc:1196] T 3fea213e5a12452f8b894b9cdb94f9ae P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fea213e5a12452f8b894b9cdb94f9ae
I20240503 08:15:13.699440 16405 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:13.700670 24098 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:13.705082 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 3fea213e5a12452f8b894b9cdb94f9ae (table rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]) successfully deleted
I20240503 08:15:13.723598 16801 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:13.735505 14880 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:13.736508 22720 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:13.752635 24125 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.753576 14878 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.759929 14349 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:13.765758 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 169bd9ea6772420abf757dd55cf44d31 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:13.824427 24129 tablet_replica.cc:330] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:13.829396 24129 raft_consensus.cc:2229] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:13.831985 24129 raft_consensus.cc:2258] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:13.834813 24129 ts_tablet_manager.cc:1875] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:13.859241 16801 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:13.864627 14880 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:13.865523 22720 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:13.869441 22720 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:13.875221 14880 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:13.885907 14349 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:13.914969 14335 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 169bd9ea6772420abf757dd55cf44d31 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:13.916694 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:13.928339 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:13.959257 16736 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 169bd9ea6772420abf757dd55cf44d31. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:13.976958 24129 ts_tablet_manager.cc:1888] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:15:13.993340 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:13.999584 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.004140 24129 log.cc:1196] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:14.005379 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 169bd9ea6772420abf757dd55cf44d31 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:14.027400 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:14.092718 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.166414 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:14.211109 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:14.212117 14619 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:14.221511 16669 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:14.222267 16405 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:14.222389 24075 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:14.223169 24075 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:14.242772 16669 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:14.250619 24199 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:14.258577 14342 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:14.259409 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:14.272384 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet 562e1ec5c34945298cf457758328057f with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48320
I20240503 08:15:14.309931 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:14.312501 24213 tablet_replica.cc:330] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:15:14.313861 24213 raft_consensus.cc:2229] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:14.314285 24213 raft_consensus.cc:2258] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:14.315595 24213 ts_tablet_manager.cc:1875] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:14.317210 16405 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:14.339296 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:14.352015 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:14.361558 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.393640 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:14.399405 24222 ts_tablet_manager.cc:902] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:15:14.399466 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:14.423519 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.455322 24222 tablet_copy_client.cc:308] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.136:37641
I20240503 08:15:14.463320 24213 ts_tablet_manager.cc:1888] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:14.464380 24213 log.cc:1196] T 562e1ec5c34945298cf457758328057f P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/562e1ec5c34945298cf457758328057f
I20240503 08:15:14.472443 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet 562e1ec5c34945298cf457758328057f (table rebalance_test_table_5 [id=d5bb4e27f85346a0953339b3915769bc]) successfully deleted
I20240503 08:15:14.497493 16821 tablet_copy_service.cc:140] P 036223b0890d4634acfadf9596f39f80: Received BeginTabletCopySession request for tablet 169bd9ea6772420abf757dd55cf44d31 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:38225)
I20240503 08:15:14.497709 16821 tablet_copy_service.cc:161] P 036223b0890d4634acfadf9596f39f80: Beginning new tablet copy session on tablet 169bd9ea6772420abf757dd55cf44d31 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:38225: session id = 6835ad958e094ce4927135b942a2e8a8-169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:14.501427 16821 tablet_copy_source_session.cc:205] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:14.505313 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:14.540282 24222 tablet_copy_client.cc:783] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:14.541594 24222 tablet_copy_client.cc:647] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:14.554821 24222 tablet_copy_client.cc:523] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:14.561658 24222 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:15:14.626493 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:14.674377 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:14.684392 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:14.684819 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:14.723049 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.725324 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:14.756425 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:14.756897 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:14.771303 24222 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:14.771797 24222 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:15:14.772091 24222 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.211s	user 0.054s	sys 0.000s
I20240503 08:15:14.772811 24222 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:14.773031 24222 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:15:14.802881 24222 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:14.829308 16669 raft_consensus.cc:1203] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:15:14.850353 24222 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.078s	user 0.003s	sys 0.000s
I20240503 08:15:14.855015 16821 tablet_copy_service.cc:342] P 036223b0890d4634acfadf9596f39f80: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-169bd9ea6772420abf757dd55cf44d31 received from {username='slave'} at 127.2.49.135:38225
I20240503 08:15:14.855245 16821 tablet_copy_service.cc:433] P 036223b0890d4634acfadf9596f39f80: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-169bd9ea6772420abf757dd55cf44d31 on tablet 169bd9ea6772420abf757dd55cf44d31 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:15:14.977136 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:14.989094 16405 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:14.990105 24282 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:15:15.040352 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 43974.3us
I20240503 08:15:15.049850 24282 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:15.060209 16405 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:15.075630 14344 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 5 to 7, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:15:15.113197 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 80132.9us
I20240503 08:15:15.156746 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:15.160266 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a949e76ab5477680a4f9912e9672e9 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45478
I20240503 08:15:15.161041 24343 tablet_replica.cc:330] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:15.161319 24343 raft_consensus.cc:2229] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:15.161581 24343 raft_consensus.cc:2258] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:15.162750 24343 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:15:15.163218 14817 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72a949e76ab5477680a4f9912e9672e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:15.166455 16404 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:15.167112 24098 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:15.170679 24343 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:15:15.170338 24098 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.170974 24343 log.cc:1196] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:15.171818 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 72a949e76ab5477680a4f9912e9672e9 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:15.175683 16405 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.178067 14334 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 72a949e76ab5477680a4f9912e9672e9 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:15.178364 14490 consensus_queue.cc:237] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:15.184681 14342 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.191192 16535 raft_consensus.cc:1261] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:15.192044 24256 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:15.196761 24352 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.200480 16535 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.203639 14348 catalog_manager.cc:5484] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.214651 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c687520c0a349fd8438dacbd5011001 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:15.215852 14490 consensus_queue.cc:237] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:15:15.219035 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c687520c0a349fd8438dacbd5011001. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:15.223007 24356 tablet_replica.cc:330] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:15.225031 16536 raft_consensus.cc:1261] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:15.226038 24352 consensus_queue.cc:1035] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:15.226833 24356 raft_consensus.cc:2229] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:15.230072 24356 raft_consensus.cc:2258] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:15.230974 24356 ts_tablet_manager.cc:1875] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:15.230662 24352 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:15.234941 16536 raft_consensus.cc:2932] T 1c687520c0a349fd8438dacbd5011001 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:15.235823 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1c687520c0a349fd8438dacbd5011001 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:15.240968 14348 catalog_manager.cc:5484] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:15.265729 24356 ts_tablet_manager.cc:1888] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:15.267907 24356 log.cc:1196] T 1c687520c0a349fd8438dacbd5011001 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c687520c0a349fd8438dacbd5011001
I20240503 08:15:15.268828 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 1c687520c0a349fd8438dacbd5011001 (table rebalance_test_table_4 [id=64eb775ebfdc4fad97f1290c92213ff7]) successfully deleted
I20240503 08:15:15.432235 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:15.482879 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:15.485163 16669 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:15.485821 24352 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:15.499552 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:15.506350 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:15.537261 24352 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:15.549763 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:15.550716 14348 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:15.552131 16669 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:15.564477 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48320
I20240503 08:15:15.564487 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
W20240503 08:15:15.571341 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da1f165b4a5647a48ad7ff951fb3fbf6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:15.600066 24441 tablet_replica.cc:330] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:15:15.600689 24441 raft_consensus.cc:2229] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:15.601042 24441 raft_consensus.cc:2258] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:15.601963 24441 ts_tablet_manager.cc:1875] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:15.603911 16667 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:15.613575 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:15.614518 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:15.615082 24442 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:15.629132 24256 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.633644 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet da1f165b4a5647a48ad7ff951fb3fbf6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:15.634487 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:15.635039 16669 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.636163 14342 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:15.642141 24441 ts_tablet_manager.cc:1888] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:15.655025 24441 log.cc:1196] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:15.656919 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet da1f165b4a5647a48ad7ff951fb3fbf6 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:15.658767 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:15.687793 24448 ts_tablet_manager.cc:902] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:15.776937 24448 tablet_copy_client.cc:308] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:15.778190 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 72a949e76ab5477680a4f9912e9672e9 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:44455)
I20240503 08:15:15.778347 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 72a949e76ab5477680a4f9912e9672e9 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:44455: session id = 95860e22c9474e2d894f04deba8442cc-72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:15.781677 14901 tablet_copy_source_session.cc:205] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:15.784116 24448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:15.816740 24476 ts_tablet_manager.cc:902] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:15.830595 24448 tablet_copy_client.cc:783] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:15.832465 24448 tablet_copy_client.cc:647] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:15.835049 24448 tablet_copy_client.cc:523] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:15.880226 24476 tablet_copy_client.cc:308] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:15.881078 24448 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:15:15.881311 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 1c687520c0a349fd8438dacbd5011001 from peer 6835ad958e094ce4927135b942a2e8a8 ({username='slave'} at 127.2.49.135:44583)
I20240503 08:15:15.881492 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 1c687520c0a349fd8438dacbd5011001 from peer 6835ad958e094ce4927135b942a2e8a8 at {username='slave'} at 127.2.49.135:44583: session id = 6835ad958e094ce4927135b942a2e8a8-1c687520c0a349fd8438dacbd5011001
I20240503 08:15:15.884676 14507 tablet_copy_source_session.cc:205] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:15.887104 24476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c687520c0a349fd8438dacbd5011001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:15.935654 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:15.984545 24511 ts_tablet_manager.cc:902] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:15.984728 24476 tablet_copy_client.cc:783] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:15.984987 24476 tablet_copy_client.cc:647] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:15.989748 24476 tablet_copy_client.cc:523] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:16.001675 24476 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap starting.
I20240503 08:15:16.023753 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:16.035669 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:16.070565 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:16.076197 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:16.114704 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:16.149931 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:16.154379 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:16.178411 24511 tablet_copy_client.cc:308] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:16.179530 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer 95860e22c9474e2d894f04deba8442cc ({username='slave'} at 127.2.49.134:50025)
I20240503 08:15:16.179769 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer 95860e22c9474e2d894f04deba8442cc at {username='slave'} at 127.2.49.134:50025: session id = 95860e22c9474e2d894f04deba8442cc-da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:16.184082 14507 tablet_copy_source_session.cc:205] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:16.192551 24511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:16.207111 24511 tablet_copy_client.cc:783] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:16.207394 24511 tablet_copy_client.cc:647] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:16.209535 24511 tablet_copy_client.cc:523] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:16.212911 24511 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: Bootstrap starting.
I20240503 08:15:16.299206 24448 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:16.299813 24448 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:15:16.300886 24448 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.420s	user 0.065s	sys 0.000s
I20240503 08:15:16.301687 24448 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:16.302341 24448 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:15:16.302603 24476 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:16.302660 24448 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:16.303547 24476 tablet_bootstrap.cc:492] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: Bootstrap complete.
I20240503 08:15:16.303838 24476 ts_tablet_manager.cc:1367] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: Time spent bootstrapping tablet: real 0.302s	user 0.060s	sys 0.002s
I20240503 08:15:16.304752 24476 raft_consensus.cc:348] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:16.304962 24476 raft_consensus.cc:729] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Initialized, Role: FOLLOWER
I20240503 08:15:16.305224 24476 consensus_queue.cc:260] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:16.314834 24448 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20240503 08:15:16.315490 24476 ts_tablet_manager.cc:1398] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.003s
I20240503 08:15:16.315816 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-72a949e76ab5477680a4f9912e9672e9 received from {username='slave'} at 127.2.49.134:44455
I20240503 08:15:16.315940 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-72a949e76ab5477680a4f9912e9672e9 on tablet 72a949e76ab5477680a4f9912e9672e9 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:15:16.316259 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 6835ad958e094ce4927135b942a2e8a8-1c687520c0a349fd8438dacbd5011001 received from {username='slave'} at 127.2.49.135:44583
I20240503 08:15:16.316393 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 6835ad958e094ce4927135b942a2e8a8-1c687520c0a349fd8438dacbd5011001 on tablet 1c687520c0a349fd8438dacbd5011001 with peer 6835ad958e094ce4927135b942a2e8a8
I20240503 08:15:16.369832 24511 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:16.370402 24511 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: Bootstrap complete.
I20240503 08:15:16.370708 24511 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: Time spent bootstrapping tablet: real 0.158s	user 0.057s	sys 0.002s
I20240503 08:15:16.371567 24511 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:16.371780 24511 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95860e22c9474e2d894f04deba8442cc, State: Initialized, Role: FOLLOWER
I20240503 08:15:16.372084 24511 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:16.372848 24511 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:15:16.376641 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 95860e22c9474e2d894f04deba8442cc-da1f165b4a5647a48ad7ff951fb3fbf6 received from {username='slave'} at 127.2.49.134:50025
I20240503 08:15:16.376824 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 95860e22c9474e2d894f04deba8442cc-da1f165b4a5647a48ad7ff951fb3fbf6 on tablet da1f165b4a5647a48ad7ff951fb3fbf6 with peer 95860e22c9474e2d894f04deba8442cc
I20240503 08:15:16.502465 14619 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:16.514261 16669 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:16.514894 24325 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:16.527890 24617 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.528563 16669 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.534502 14344 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.544643 16535 raft_consensus.cc:1203] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:16.611860 14619 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:16.614496 16669 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:16.616014 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:16.616359 24325 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:16.618531 16669 raft_consensus.cc:1203] T 1c687520c0a349fd8438dacbd5011001 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:16.619243 14749 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:15:16.620509 24617 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:16.622892 24618 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:16.626348 14749 raft_consensus.cc:858] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
W20240503 08:15:16.626492 14749 raft_consensus.cc:1565] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 NON_PARTICIPANT]: Could not prepare op 'term: 2 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:16.626519 16667 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:16.631495 14333 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4e86f4f3a01048e69f553c6557d79b71 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:16.637928 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:16.642956 16535 raft_consensus.cc:1203] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:15:16.656603 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:16.708340 14749 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 NON_PARTICIPANT]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.715301 14619 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:16.717604 16667 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:16.718416 24617 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:16.723872 24618 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.724484 16667 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.725175 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:16.729887 14344 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:16.737995 14619 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:16.739670 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:16.740657 14555 consensus_peers.cc:485] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7a5221747794a13aa4ed763006ba291. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:16.743366 16667 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:16.744211 24617 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:16.750425 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a5221747794a13aa4ed763006ba291 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:16.752120 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:16.757020 24617 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:16.759543 16669 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:15:16.760397 14336 catalog_manager.cc:4804] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): delete failed for tablet 4e86f4f3a01048e69f553c6557d79b71 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 6 but the committed config has opid_index of 7
I20240503 08:15:16.761086 24641 tablet_replica.cc:330] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:16.761287 24641 raft_consensus.cc:2229] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:16.761480 24641 raft_consensus.cc:2258] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:16.762355 24641 ts_tablet_manager.cc:1875] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:16.769743 14748 raft_consensus.cc:1467] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 08:15:16.770560 14748 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:16.777884 14346 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:16.787945 24641 ts_tablet_manager.cc:1888] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:16.788098 24641 log.cc:1196] T f7a5221747794a13aa4ed763006ba291 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/f7a5221747794a13aa4ed763006ba291
I20240503 08:15:16.788931 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet f7a5221747794a13aa4ed763006ba291 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:16.833577 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:16.834372 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:16.862454 14333 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f7a5221747794a13aa4ed763006ba291 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:16.880143 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:16.915189 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:17.216375 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:17.222741 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:17.235471 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:17.236105 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:17.297742 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:17.306224 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:17.308744 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:17.317173 24753 ts_tablet_manager.cc:902] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:15:17.335261 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:17.414888 24753 tablet_copy_client.cc:308] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:15:17.415670 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet f7a5221747794a13aa4ed763006ba291 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:53581)
I20240503 08:15:17.415870 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet f7a5221747794a13aa4ed763006ba291 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:53581: session id = 0f8702fad8ea4d22840dab11bb27f7f2-f7a5221747794a13aa4ed763006ba291
I20240503 08:15:17.419490 14641 tablet_copy_source_session.cc:205] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:17.420939 14748 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:17.424247 24753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a5221747794a13aa4ed763006ba291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:17.452082 16535 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:17.453612 16801 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:17.454588 24316 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:17.455221 24316 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:15:17.456466 14687 consensus_peers.cc:485] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:17.457330 24753 tablet_copy_client.cc:783] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:17.457602 24753 tablet_copy_client.cc:647] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:17.461011 24753 tablet_copy_client.cc:523] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:17.469652 24753 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:15:17.492046 24777 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:17.498289 16800 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:17.499286 16535 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:17.500867 14348 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:17.667543 24753 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:17.668027 24753 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:15:17.668337 24753 ts_tablet_manager.cc:1367] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.199s	user 0.060s	sys 0.008s
I20240503 08:15:17.669176 24753 raft_consensus.cc:348] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:17.669422 24753 raft_consensus.cc:729] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:15:17.670125 24753 consensus_queue.cc:260] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:17.690582 24753 ts_tablet_manager.cc:1398] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.000s
I20240503 08:15:17.692907 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:17.693243 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-f7a5221747794a13aa4ed763006ba291 received from {username='slave'} at 127.2.49.133:53581
I20240503 08:15:17.693373 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-f7a5221747794a13aa4ed763006ba291 on tablet f7a5221747794a13aa4ed763006ba291 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:15:17.706482 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:17.728276 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:17.754072 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:17.755189 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:17.767031 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:17.775043 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:17.819733 16800 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:17.851770 14880 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:15:17.852897 24153 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:17.851630 16669 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 08:15:17.854473 16737 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:17.855101 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:17.856393 24153 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:17.860580 16404 raft_consensus.cc:1203] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:17.895833 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:17.916779 24858 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:17.924965 16669 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:17.925704 14880 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:17.933220 14346 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:17.954339 14599 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:17.987329 24868 ts_tablet_manager.cc:902] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:15:18.044039 14490 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:18.052435 16669 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:18.053073 24852 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:18.060171 24868 tablet_copy_client.cc:308] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:15:18.060585 24852 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:18.062520 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet 897a701b880641b08816f2d1a254ef03 from peer 129a2ac87d7f416db6d47d79bb1c2b5f ({username='slave'} at 127.2.49.129:59689)
I20240503 08:15:18.062685 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet 897a701b880641b08816f2d1a254ef03 from peer 129a2ac87d7f416db6d47d79bb1c2b5f at {username='slave'} at 127.2.49.129:59689: session id = 129a2ac87d7f416db6d47d79bb1c2b5f-897a701b880641b08816f2d1a254ef03
I20240503 08:15:18.064848 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:18.065995 14771 tablet_copy_source_session.cc:205] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:18.072708 14346 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:18.076360 24868 ts_tablet_manager.cc:1875] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:18.079001 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:18.084847 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b13cdf4b234f17872f5d1588915968 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:18.092264 14490 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:18.104936 24908 tablet_replica.cc:330] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:18.105366 24908 raft_consensus.cc:2229] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:18.105737 24908 raft_consensus.cc:2258] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:18.106616 24908 ts_tablet_manager.cc:1875] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:15:18.109606 14426 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30b13cdf4b234f17872f5d1588915968. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:18.110143 16649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:18.111140 16667 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:18.111752 24852 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:18.112867 16668 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:18.117986 24868 ts_tablet_manager.cc:1888] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:15:18.118345 24868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897a701b880641b08816f2d1a254ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:18.127140 24852 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:18.131054 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b13cdf4b234f17872f5d1588915968 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:18.135354 16668 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:18.143323 14346 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 6 to 7, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:18.148716 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:18.166847 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:18.171267 24868 tablet_copy_client.cc:783] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:18.171780 24868 tablet_copy_client.cc:647] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:18.174208 24868 tablet_copy_client.cc:523] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:18.178783 24868 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
W20240503 08:15:18.197930 14816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 43672.6us
I20240503 08:15:18.200353 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:18.252224 14880 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:18.273720 24908 ts_tablet_manager.cc:1888] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:15:18.273897 24908 log.cc:1196] T 30b13cdf4b234f17872f5d1588915968 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/30b13cdf4b234f17872f5d1588915968
I20240503 08:15:18.277673 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 30b13cdf4b234f17872f5d1588915968 (table rebalance_test_table_2 [id=f9617d8cfb394c7d8239ea6325ea98db]) successfully deleted
W20240503 08:15:18.278647 14814 consensus_peers.cc:485] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:18.279894 16404 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:18.280540 24098 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:18.285516 24868 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:18.295883 16800 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:18.296769 24098 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:18.302763 24868 tablet_bootstrap.cc:492] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap complete.
I20240503 08:15:18.303124 24868 ts_tablet_manager.cc:1367] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.124s	user 0.069s	sys 0.000s
I20240503 08:15:18.307266 24868 raft_consensus.cc:348] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:18.307543 24868 raft_consensus.cc:729] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:15:18.307787 24868 consensus_queue.cc:260] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:18.308503 24868 ts_tablet_manager.cc:1398] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 08:15:18.309331 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-897a701b880641b08816f2d1a254ef03 received from {username='slave'} at 127.2.49.129:59689
I20240503 08:15:18.309496 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-897a701b880641b08816f2d1a254ef03 on tablet 897a701b880641b08816f2d1a254ef03 with peer 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:15:18.328567 24098 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:18.337550 14346 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:18.345527 14619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71"
dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54238
I20240503 08:15:18.345746 14619 raft_consensus.cc:595] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:15:18.348706 16800 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:18.348241 16404 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:18.381816 24618 raft_consensus.cc:979] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: : Instructing follower 6835ad958e094ce4927135b942a2e8a8 to start an election
I20240503 08:15:18.382051 24618 raft_consensus.cc:1067] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Signalling peer 6835ad958e094ce4927135b942a2e8a8 to start an election
I20240503 08:15:18.383011 16667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71"
dest_uuid: "6835ad958e094ce4927135b942a2e8a8"
 from {username='slave'} at 127.2.49.130:51523
I20240503 08:15:18.383383 16667 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:18.383525 16667 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:18.391579 24949 ts_tablet_manager.cc:902] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Initiating tablet copy from peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:15:18.391469 16667 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:18.397382 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "6835ad958e094ce4927135b942a2e8a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:15:18.397624 16667 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [CANDIDATE]: Term 3 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:15:18.397707 14619 raft_consensus.cc:3032] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:18.397815 14619 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:15:18.398195 14619 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:18.398694 14619 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:18.402957 14619 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6835ad958e094ce4927135b942a2e8a8 in term 3.
I20240503 08:15:18.403702 16602 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 6835ad958e094ce4927135b942a2e8a8; no voters: 
I20240503 08:15:18.404225 24870 raft_consensus.cc:2781] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:18.405292 14748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "6835ad958e094ce4927135b942a2e8a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:15:18.405589 14748 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:18.406469 24870 raft_consensus.cc:686] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Becoming Leader. State: Replica: 6835ad958e094ce4927135b942a2e8a8, State: Running, Role: LEADER
I20240503 08:15:18.407058 24870 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:18.415647 14490 raft_consensus.cc:1203] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:18.417416 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: term changed from 2 to 3, leader changed from 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) to 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135). New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:18.468364 14748 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6835ad958e094ce4927135b942a2e8a8 in term 3.
I20240503 08:15:18.505474 24949 tablet_copy_client.cc:308] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.136:37641
I20240503 08:15:18.506611 16821 tablet_copy_service.cc:140] P 036223b0890d4634acfadf9596f39f80: Received BeginTabletCopySession request for tablet 169bd9ea6772420abf757dd55cf44d31 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 ({username='slave'} at 127.2.49.131:38253)
I20240503 08:15:18.506789 16821 tablet_copy_service.cc:161] P 036223b0890d4634acfadf9596f39f80: Beginning new tablet copy session on tablet 169bd9ea6772420abf757dd55cf44d31 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 at {username='slave'} at 127.2.49.131:38253: session id = c0fd1ebec9da48d4b0c1ae0f0ee96452-169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:18.509676 16821 tablet_copy_source_session.cc:205] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:18.516614 24949 ts_tablet_manager.cc:1875] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:18.553726 24995 ts_tablet_manager.cc:902] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:18.563795 24949 ts_tablet_manager.cc:1888] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 08:15:18.621001 24949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:18.670508 24949 tablet_copy_client.cc:783] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:18.671195 24949 tablet_copy_client.cc:647] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:18.673964 24949 tablet_copy_client.cc:523] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:18.696285 24995 tablet_copy_client.cc:308] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:18.697127 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 30b13cdf4b234f17872f5d1588915968 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 ({username='slave'} at 127.2.49.131:58129)
I20240503 08:15:18.697362 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 30b13cdf4b234f17872f5d1588915968 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 at {username='slave'} at 127.2.49.131:58129: session id = c0fd1ebec9da48d4b0c1ae0f0ee96452-30b13cdf4b234f17872f5d1588915968
I20240503 08:15:18.703616 14507 tablet_copy_source_session.cc:205] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:18.705638 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b13cdf4b234f17872f5d1588915968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:18.720824 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:18.724290 24949 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
W20240503 08:15:18.734993 14424 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): Couldn't send request to peer b35c4c5d528c4a8f9e754ff9481c3938. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:18.735863 16536 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:18.736544 24912 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:18.739511 16667 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:18.740942 24905 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:18.756028 24905 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:18.759820 16668 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:18.770361 14348 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:18.777256 16536 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
W20240503 08:15:18.814123 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 77811.2us
I20240503 08:15:18.841013 14748 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 08:15:18.842567 24995 tablet_copy_client.cc:783] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:18.842867 24995 tablet_copy_client.cc:647] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:18.846347 24870 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:18.847579 24995 tablet_copy_client.cc:523] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:18.855655 24995 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:15:18.858023 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:18.865828 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:18.923786 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:18.925274 25052 ts_tablet_manager.cc:902] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:18.978453 24995 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:18.979079 24995 tablet_bootstrap.cc:492] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap complete.
I20240503 08:15:18.979552 24995 ts_tablet_manager.cc:1367] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.124s	user 0.067s	sys 0.001s
I20240503 08:15:18.980887 24995 raft_consensus.cc:348] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:18.981191 24995 raft_consensus.cc:729] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:15:18.981514 24995 consensus_queue.cc:260] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:18.982514 24995 ts_tablet_manager.cc:1398] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:15:18.983400 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-30b13cdf4b234f17872f5d1588915968 received from {username='slave'} at 127.2.49.131:58129
I20240503 08:15:18.983542 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-30b13cdf4b234f17872f5d1588915968 on tablet 30b13cdf4b234f17872f5d1588915968 with peer c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:15:18.986625 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:18.988822 25052 tablet_copy_client.cc:308] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:18.989737 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 88404b9cdf574cb0bca1bb49348fb4cd from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:33947)
I20240503 08:15:18.989905 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 88404b9cdf574cb0bca1bb49348fb4cd from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:33947: session id = 0d77c4b1f12049dea71b97e86be2b288-88404b9cdf574cb0bca1bb49348fb4cd
I20240503 08:15:18.993147 14901 tablet_copy_source_session.cc:205] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:18.994555 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:18.998332 25052 ts_tablet_manager.cc:1875] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:19.001192 14619 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 08:15:19.001818 24870 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:19.022089 14748 raft_consensus.cc:1203] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:15:19.077715 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:19.100831 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:19.102926 25052 ts_tablet_manager.cc:1888] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:15:19.103307 25052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88404b9cdf574cb0bca1bb49348fb4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:19.131249 25088 ts_tablet_manager.cc:902] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:19.151412 24949 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:19.152058 24949 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap complete.
I20240503 08:15:19.152416 24949 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.428s	user 0.060s	sys 0.005s
I20240503 08:15:19.153544 24949 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:19.153877 24949 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:15:19.161206 24949 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:19.165235 24949 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.001s
I20240503 08:15:19.167299 16821 tablet_copy_service.cc:342] P 036223b0890d4634acfadf9596f39f80: Request end of tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-169bd9ea6772420abf757dd55cf44d31 received from {username='slave'} at 127.2.49.131:38253
I20240503 08:15:19.167462 16821 tablet_copy_service.cc:433] P 036223b0890d4634acfadf9596f39f80: ending tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-169bd9ea6772420abf757dd55cf44d31 on tablet 169bd9ea6772420abf757dd55cf44d31 with peer c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:15:19.200275 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:19.196023 25052 tablet_copy_client.cc:783] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:19.201668 25052 tablet_copy_client.cc:647] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:19.206887 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:19.213943 25052 tablet_copy_client.cc:523] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:19.217742 25088 tablet_copy_client.cc:308] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:19.218645 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer b35c4c5d528c4a8f9e754ff9481c3938 ({username='slave'} at 127.2.49.132:36825)
I20240503 08:15:19.218822 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer b35c4c5d528c4a8f9e754ff9481c3938 at {username='slave'} at 127.2.49.132:36825: session id = b35c4c5d528c4a8f9e754ff9481c3938-da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:19.221697 14507 tablet_copy_source_session.cc:205] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:19.223357 25088 ts_tablet_manager.cc:1875] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:19.224740 16404 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:15:19.225330 24098 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:15:19.227154 14817 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:19.231276 25088 ts_tablet_manager.cc:1888] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:15:19.231670 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:19.235061 16536 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:15:19.235702 24098 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:19.244123 25052 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:15:19.235702 25088 tablet_copy_client.cc:783] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:19.251906 25088 tablet_copy_client.cc:647] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:19.259783 25088 tablet_copy_client.cc:523] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:19.265358 25088 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:15:19.300508 25112 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:19.302597 16535 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:19.325086 14748 raft_consensus.cc:1203] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 08:15:19.327389 14341 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 8 to 9, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:19.337980 16404 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:19.485860 25052 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:19.487310 25052 tablet_bootstrap.cc:492] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:19.488915 25052 ts_tablet_manager.cc:1367] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.245s	user 0.063s	sys 0.006s
I20240503 08:15:19.500255 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:19.520788 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:19.549341 25052 raft_consensus.cc:348] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:19.553912 25052 raft_consensus.cc:729] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:19.554323 25052 consensus_queue.cc:260] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:19.555790 25052 ts_tablet_manager.cc:1398] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.066s	user 0.000s	sys 0.002s
I20240503 08:15:19.566278 16667 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:19.567605 14468 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:19.569180 14619 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 08:15:19.570899 25075 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:19.574618 25075 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:19.577431 14619 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:19.587950 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-88404b9cdf574cb0bca1bb49348fb4cd received from {username='slave'} at 127.2.49.130:33947
I20240503 08:15:19.588119 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-88404b9cdf574cb0bca1bb49348fb4cd on tablet 88404b9cdf574cb0bca1bb49348fb4cd with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:19.589735 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 8 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:19.591452 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:19.592291 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:19.594338 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:19.596017 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:19.603446 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:19.637310 14731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:19.651846 25162 tablet_replica.cc:330] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:19.652220 25162 raft_consensus.cc:2229] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:19.652572 25162 raft_consensus.cc:2258] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:19.653810 25162 ts_tablet_manager.cc:1875] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:19.722303 25088 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:19.722795 25162 ts_tablet_manager.cc:1888] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240503 08:15:19.722862 25088 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap complete.
I20240503 08:15:19.723145 25088 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.458s	user 0.092s	sys 0.021s
I20240503 08:15:19.727159 25088 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:19.727478 25088 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:15:19.728966 25088 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:19.729522 25162 log.cc:1196] T 4e86f4f3a01048e69f553c6557d79b71 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:19.730481 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet 4e86f4f3a01048e69f553c6557d79b71 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:19.731509 25088 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240503 08:15:19.732218 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-da1f165b4a5647a48ad7ff951fb3fbf6 received from {username='slave'} at 127.2.49.132:36825
I20240503 08:15:19.732324 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-da1f165b4a5647a48ad7ff951fb3fbf6 on tablet da1f165b4a5647a48ad7ff951fb3fbf6 with peer b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:15:19.763134 16667 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:19.766413 14619 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:15:19.767033 25064 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:15:19.769613 16603 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e86f4f3a01048e69f553c6557d79b71. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:19.770381 25064 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:19.776131 14619 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:19.776968 14335 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4e86f4f3a01048e69f553c6557d79b71 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:19.777999 14621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028"
dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54298
I20240503 08:15:19.778204 14621 raft_consensus.cc:595] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:15:19.783093 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 10 to 11, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:19.788537 14880 consensus_queue.cc:237] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:19.807413 16800 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:19.807502 25189 ts_tablet_manager.cc:902] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
W20240503 08:15:19.807679 14817 consensus_peers.cc:485] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:19.809540 25127 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:19.816303 14619 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:15:19.817623 25127 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:19.848407 25112 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:19.849299 16800 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:19.854275 14619 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:19.855744 14348 catalog_manager.cc:5484] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 5 to 6, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:19.907675 14619 raft_consensus.cc:1203] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:19.923342 25189 tablet_copy_client.cc:308] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:19.924474 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 72a949e76ab5477680a4f9912e9672e9 from peer 129a2ac87d7f416db6d47d79bb1c2b5f ({username='slave'} at 127.2.49.129:53921)
I20240503 08:15:19.924687 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 72a949e76ab5477680a4f9912e9672e9 from peer 129a2ac87d7f416db6d47d79bb1c2b5f at {username='slave'} at 127.2.49.129:53921: session id = 129a2ac87d7f416db6d47d79bb1c2b5f-72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:19.928756 14901 tablet_copy_source_session.cc:205] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:19.932514 25189 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:19.939412 14880 raft_consensus.cc:1203] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:19.940016 25189 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 08:15:19.940338 25189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:19.947198 25189 tablet_copy_client.cc:783] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:19.947536 25189 tablet_copy_client.cc:647] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:19.949417 25189 tablet_copy_client.cc:523] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:19.955472 25189 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:15:20.103364 24851 raft_consensus.cc:979] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: : Instructing follower 129a2ac87d7f416db6d47d79bb1c2b5f to start an election
I20240503 08:15:20.103572 25246 raft_consensus.cc:1067] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Signalling peer 129a2ac87d7f416db6d47d79bb1c2b5f to start an election
I20240503 08:15:20.104399 14490 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028"
dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
 from {username='slave'} at 127.2.49.130:47845
I20240503 08:15:20.104662 14490 raft_consensus.cc:482] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:20.104763 14490 raft_consensus.cc:3037] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:20.108500 14490 raft_consensus.cc:504] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:20.110204 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:15:20.110883 14490 leader_election.cc:288] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 2 election: Requested vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:15:20.111275 16405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd11460051ab40febc2fe6aff58b5028" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0f8702fad8ea4d22840dab11bb27f7f2"
I20240503 08:15:20.111506 16405 raft_consensus.cc:3037] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:20.111553 14619 raft_consensus.cc:3032] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:15:20.111644 14619 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:15:20.111897 25189 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:20.111913 14619 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:20.112254 25189 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap complete.
I20240503 08:15:20.112371 14619 raft_consensus.cc:3037] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:15:20.112493 25189 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.157s	user 0.059s	sys 0.008s
I20240503 08:15:20.113164 25189 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:20.113379 25189 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:15:20.113590 25189 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:20.114112 25189 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:15:20.115739 16405 raft_consensus.cc:2454] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 2.
I20240503 08:15:20.115855 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-72a949e76ab5477680a4f9912e9672e9 received from {username='slave'} at 127.2.49.129:53921
I20240503 08:15:20.116007 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-72a949e76ab5477680a4f9912e9672e9 on tablet 72a949e76ab5477680a4f9912e9672e9 with peer 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:15:20.116127 14619 raft_consensus.cc:2454] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a2ac87d7f416db6d47d79bb1c2b5f in term 2.
I20240503 08:15:20.117027 14423 leader_election.cc:302] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 
I20240503 08:15:20.117331 25198 raft_consensus.cc:2781] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:15:20.117436 25198 raft_consensus.cc:686] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Becoming Leader. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Running, Role: LEADER
I20240503 08:15:20.117673 25198 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:20.132377 14341 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: term changed from 1 to 2, leader changed from 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) to 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129). New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:20.227186 14490 raft_consensus.cc:1203] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 08:15:20.310325 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.319265 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:20.357688 25303 ts_tablet_manager.cc:902] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:15:20.374563 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.393007 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:20.395857 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:20.404947 25312 ts_tablet_manager.cc:902] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:20.406971 25303 tablet_copy_client.cc:308] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.135:46299
I20240503 08:15:20.409793 16689 tablet_copy_service.cc:140] P 6835ad958e094ce4927135b942a2e8a8: Received BeginTabletCopySession request for tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:41763)
I20240503 08:15:20.410007 16689 tablet_copy_service.cc:161] P 6835ad958e094ce4927135b942a2e8a8: Beginning new tablet copy session on tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:41763: session id = 0f8702fad8ea4d22840dab11bb27f7f2-4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:20.413983 16689 tablet_copy_source_session.cc:205] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:20.416363 25303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:20.433745 25303 tablet_copy_client.cc:783] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:20.464568 25303 tablet_copy_client.cc:647] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:20.477231 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.479568 25312 tablet_copy_client.cc:308] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:20.480405 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 0780fb979eee4bae98a014d9721c9441 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 ({username='slave'} at 127.2.49.131:56291)
I20240503 08:15:20.480584 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 0780fb979eee4bae98a014d9721c9441 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 at {username='slave'} at 127.2.49.131:56291: session id = c0fd1ebec9da48d4b0c1ae0f0ee96452-0780fb979eee4bae98a014d9721c9441
I20240503 08:15:20.483439 14901 tablet_copy_source_session.cc:205] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:20.487298 25312 ts_tablet_manager.cc:1875] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:20.489305 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:20.506232 25303 tablet_copy_client.cc:523] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:20.511750 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:20.544952 25303 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:15:20.574735 25312 ts_tablet_manager.cc:1888] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:15:20.578333 25312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0780fb979eee4bae98a014d9721c9441. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:20.594497 25312 tablet_copy_client.cc:783] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:20.594825 25312 tablet_copy_client.cc:647] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:20.601079 25312 tablet_copy_client.cc:523] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:20.627699 25312 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:15:20.632128 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.642740 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.663859 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:20.669450 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:20.670404 16405 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:20.670961 25198 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:20.671202 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:20.716205 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.726613 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:20.733510 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:20.748132 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:20.759552 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:20.767179 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.770368 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.816669 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:20.817955 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:20.827620 14619 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:15:20.829192 25377 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:15:20.916416 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:20.944633 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:21.081482 25303 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:21.081909 25303 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:15:21.082970 25303 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.538s	user 0.052s	sys 0.008s
I20240503 08:15:21.083703 25303 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:21.083928 25303 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:15:21.085526 25303 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:21.086062 25303 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:15:21.086846 16689 tablet_copy_service.cc:342] P 6835ad958e094ce4927135b942a2e8a8: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-4e86f4f3a01048e69f553c6557d79b71 received from {username='slave'} at 127.2.49.133:41763
I20240503 08:15:21.086983 16689 tablet_copy_service.cc:433] P 6835ad958e094ce4927135b942a2e8a8: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-4e86f4f3a01048e69f553c6557d79b71 on tablet 4e86f4f3a01048e69f553c6557d79b71 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:15:21.121801 25312 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:21.122326 25312 tablet_bootstrap.cc:492] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap complete.
I20240503 08:15:21.122612 25312 ts_tablet_manager.cc:1367] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.495s	user 0.047s	sys 0.010s
I20240503 08:15:21.123409 25312 raft_consensus.cc:348] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:21.123665 25312 raft_consensus.cc:729] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:15:21.123943 25312 consensus_queue.cc:260] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:21.124645 25312 ts_tablet_manager.cc:1398] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:15:21.125473 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-0780fb979eee4bae98a014d9721c9441 received from {username='slave'} at 127.2.49.131:56291
I20240503 08:15:21.125595 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-0780fb979eee4bae98a014d9721c9441 on tablet 0780fb979eee4bae98a014d9721c9441 with peer c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:15:21.184862 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:21.226297 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:21.262182 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:21.274950 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:21.277094 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:21.329521 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:21.343387 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:21.389541 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:21.403025 16404 raft_consensus.cc:1203] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11]   Dedup: 3.11->[]
I20240503 08:15:21.464088 14748 raft_consensus.cc:1203] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20240503 08:15:21.548792 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:21.566684 14490 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:21.567981 16404 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:21.568882 25451 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:21.571597 25451 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:21.586297 25344 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:21.587486 16404 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:21.596570 14490 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:21.602845 14346 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:21.695258 16516 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a949e76ab5477680a4f9912e9672e9 with delete_type TABLET_DATA_TOMBSTONED (TS 95860e22c9474e2d894f04deba8442cc not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43672
I20240503 08:15:21.696280 25513 tablet_replica.cc:330] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: stopping tablet replica
I20240503 08:15:21.696955 25513 raft_consensus.cc:2229] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:21.698117 25513 raft_consensus.cc:2258] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:21.701910 25513 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:21.780908 25513 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:15:21.782387 25513 log.cc:1196] T 72a949e76ab5477680a4f9912e9672e9 P 95860e22c9474e2d894f04deba8442cc: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/wals/72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:21.783751 14336 catalog_manager.cc:4830] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): tablet 72a949e76ab5477680a4f9912e9672e9 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:21.832015 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:21.857000 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:21.870626 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:21.878492 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:21.883340 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:21.942726 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:21.943223 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:21.975091 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.542521 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.604169 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:22.610708 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.650557 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:22.677536 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.738476 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:22.741432 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.791533 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.792652 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:22.825488 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 116386us
I20240503 08:15:22.835638 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:22.845887 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:22.854591 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:22.901379 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:22.918119 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:22.959115 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:23.004153 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:23.118110 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:23.133383 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:23.175211 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:23.221101 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:23.225436 14860 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:23.277809 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:23.304746 14730 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:23.305320 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:23.345952 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:23.375357 16536 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:23.375739 16668 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:23.376040 25557 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:23.376353 25377 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:23.377625 14748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a"
dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:50116
I20240503 08:15:23.377885 14748 raft_consensus.cc:595] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:15:23.414299 25377 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:23.418929 16535 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:23.439340 16668 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:23.447702 14341 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:23.495493 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 with delete_type TABLET_DATA_TOMBSTONED (TS b35c4c5d528c4a8f9e754ff9481c3938 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48320
I20240503 08:15:23.504954 25740 tablet_replica.cc:330] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: stopping tablet replica
I20240503 08:15:23.505337 25740 raft_consensus.cc:2229] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:23.505573 25740 raft_consensus.cc:2258] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:23.506668 25740 ts_tablet_manager.cc:1875] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:23.514612 25740 ts_tablet_manager.cc:1888] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:23.514832 25740 log.cc:1196] T da1f165b4a5647a48ad7ff951fb3fbf6 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/wals/da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:23.515699 14334 catalog_manager.cc:4830] TS b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): tablet da1f165b4a5647a48ad7ff951fb3fbf6 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:23.578814 14880 consensus_queue.cc:237] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:23.584975 16801 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:23.585589 25344 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:23.585664 14748 raft_consensus.cc:1261] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:15:23.587028 25344 consensus_queue.cc:1035] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:23.599112 25344 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:23.603087 14749 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:23.607453 16800 raft_consensus.cc:2932] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:23.615517 14344 catalog_manager.cc:5484] T 0780fb979eee4bae98a014d9721c9441 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 6 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:23.620992 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 0780fb979eee4bae98a014d9721c9441 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:23.624331 25769 tablet_replica.cc:330] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:23.625087 25769 raft_consensus.cc:2229] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:23.626045 25769 raft_consensus.cc:2258] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:23.627914 25769 ts_tablet_manager.cc:1875] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:23.660895 25769 ts_tablet_manager.cc:1888] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:15:23.663097 25769 log.cc:1196] T 0780fb979eee4bae98a014d9721c9441 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/0780fb979eee4bae98a014d9721c9441
I20240503 08:15:23.664523 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 0780fb979eee4bae98a014d9721c9441 (table rebalance_test_table_3 [id=3d6b5b0c5d2848d989f291fdcbf71d6f]) successfully deleted
I20240503 08:15:23.678185 25657 raft_consensus.cc:979] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: : Instructing follower b35c4c5d528c4a8f9e754ff9481c3938 to start an election
I20240503 08:15:23.678407 25657 raft_consensus.cc:1067] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 LEADER]: Signalling peer b35c4c5d528c4a8f9e754ff9481c3938 to start an election
I20240503 08:15:23.679271 14878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a"
dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
 from {username='slave'} at 127.2.49.131:56291
I20240503 08:15:23.679457 14878 raft_consensus.cc:482] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:23.679558 14878 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:23.682672 14878 raft_consensus.cc:504] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:23.683912 14878 leader_election.cc:288] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 3 election: Requested vote from peers c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:15:23.683939 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:15:23.684228 14749 raft_consensus.cc:3032] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:23.684314 14749 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Running, Role: LEADER
I20240503 08:15:23.684671 14749 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:23.685227 14749 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:23.687347 16668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9575f64eb741948e51ccf06f48228a" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "6835ad958e094ce4927135b942a2e8a8"
I20240503 08:15:23.687605 16668 raft_consensus.cc:3037] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:23.690516 16668 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 3.
I20240503 08:15:23.691123 14816 leader_election.cc:302] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6835ad958e094ce4927135b942a2e8a8, b35c4c5d528c4a8f9e754ff9481c3938; no voters: 
I20240503 08:15:23.691515 25767 raft_consensus.cc:2781] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:23.691653 25767 raft_consensus.cc:686] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:15:23.691962 25767 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:23.710081 14749 raft_consensus.cc:2454] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 3.
I20240503 08:15:23.722496 14344 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 2 to 3, leader changed from c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:23.856542 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:23.965394 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:23.971674 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:23.975819 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:23.982879 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:23.997727 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:24.086843 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:24.132680 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:24.171314 14749 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:15:24.171967 25344 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:24.186892 14878 consensus_queue.cc:237] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:24.191215 14619 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:24.191355 16801 raft_consensus.cc:1261] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:24.191936 25767 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:24.192456 25344 consensus_queue.cc:1035] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:24.205076 25767 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:24.221899 14619 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:24.232044 14348 catalog_manager.cc:5484] T 88404b9cdf574cb0bca1bb49348fb4cd P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:24.244854 16668 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:15:24.245591 25872 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:24.271811 16801 raft_consensus.cc:2932] T 88404b9cdf574cb0bca1bb49348fb4cd P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:24.290484 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet 88404b9cdf574cb0bca1bb49348fb4cd with delete_type TABLET_DATA_TOMBSTONED (TS 0f8702fad8ea4d22840dab11bb27f7f2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43360
I20240503 08:15:24.302898 25889 tablet_replica.cc:330] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: stopping tablet replica
I20240503 08:15:24.303870 25889 raft_consensus.cc:2229] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:24.304350 25889 raft_consensus.cc:2258] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:24.307273 25889 ts_tablet_manager.cc:1875] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:24.341012 25889 ts_tablet_manager.cc:1888] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:24.363126 25889 log.cc:1196] T 88404b9cdf574cb0bca1bb49348fb4cd P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/wals/88404b9cdf574cb0bca1bb49348fb4cd
I20240503 08:15:24.364267 14334 catalog_manager.cc:4830] TS 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): tablet 88404b9cdf574cb0bca1bb49348fb4cd (table rebalance_test_table_9 [id=b120ce671eaf40d099d0c2d7e71d6cbe]) successfully deleted
I20240503 08:15:24.525065 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:24.555454 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:24.606727 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:24.628947 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:24.649418 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:24.650990 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:24.653002 16405 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:15:24.654834 25344 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:24.688617 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:24.691546 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:24.703801 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:24.703960 25871 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:24.704686 16404 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:24.709647 14340 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 10 to 11, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) evicted. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:24.715617 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:24.717768 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a949e76ab5477680a4f9912e9672e9 with delete_type TABLET_DATA_TOMBSTONED (TS 129a2ac87d7f416db6d47d79bb1c2b5f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45478
W20240503 08:15:24.718500 14816 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72a949e76ab5477680a4f9912e9672e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:24.721153 16404 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:15:24.721925 25871 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:24.741456 25964 tablet_replica.cc:330] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: stopping tablet replica
I20240503 08:15:24.744406 25964 raft_consensus.cc:2229] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:24.744521 25344 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:24.745254 25964 raft_consensus.cc:2258] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:24.747584 25964 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:24.750995 14334 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 72a949e76ab5477680a4f9912e9672e9 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:24.751104 16404 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:24.754102 14344 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 11 to 12, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:24.772933 25964 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:15:24.774272 25964 log.cc:1196] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:24.775182 14336 catalog_manager.cc:4830] TS 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): tablet 72a949e76ab5477680a4f9912e9672e9 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:25.056136 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.070839 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.084614 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:25.123436 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:25.124997 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.136066 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:25.169235 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.191514 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.235256 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:25.257247 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:25.262032 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:25.272011 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:25.296936 26039 ts_tablet_manager.cc:902] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:25.302942 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:25.306874 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.322090 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:25.343333 26039 tablet_copy_client.cc:308] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:25.344291 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 72a949e76ab5477680a4f9912e9672e9 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:41491)
I20240503 08:15:25.344435 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 72a949e76ab5477680a4f9912e9672e9 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:41491: session id = 036223b0890d4634acfadf9596f39f80-72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:25.347260 14901 tablet_copy_source_session.cc:205] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:25.348631 26039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:25.372802 26039 tablet_copy_client.cc:783] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:25.373088 26039 tablet_copy_client.cc:647] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:25.380003 26039 tablet_copy_client.cc:523] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:25.396435 26039 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:15:25.419097 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:25.513221 26039 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:25.513685 26039 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:15:25.513949 26039 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.118s	user 0.051s	sys 0.001s
I20240503 08:15:25.514997 26039 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:25.515206 26039 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:15:25.515484 26039 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:25.516192 26039 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.002s	user 0.006s	sys 0.000s
I20240503 08:15:25.516933 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-72a949e76ab5477680a4f9912e9672e9 received from {username='slave'} at 127.2.49.136:41491
I20240503 08:15:25.517046 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 036223b0890d4634acfadf9596f39f80-72a949e76ab5477680a4f9912e9672e9 on tablet 72a949e76ab5477680a4f9912e9672e9 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:15:25.662365 14880 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:25.675438 16667 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 08:15:25.678642 26050 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:25.739069 26050 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:25.743711 16668 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:25.751624 14348 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 6 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) evicted. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:25.774667 16801 raft_consensus.cc:1203] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
I20240503 08:15:25.803260 14730 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9575f64eb741948e51ccf06f48228a with delete_type TABLET_DATA_TOMBSTONED (TS c0fd1ebec9da48d4b0c1ae0f0ee96452 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44098
I20240503 08:15:25.803457 14880 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
W20240503 08:15:25.806571 14814 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc9575f64eb741948e51ccf06f48228a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:25.807498 16667 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 08:15:25.809480 25874 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:25.809779 26107 tablet_replica.cc:330] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: stopping tablet replica
I20240503 08:15:25.809981 26107 raft_consensus.cc:2229] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:25.816632 25874 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:25.821080 16667 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:25.831724 14334 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dc9575f64eb741948e51ccf06f48228a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:25.835520 14344 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:25.810149 26107 raft_consensus.cc:2258] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:25.865624 26107 ts_tablet_manager.cc:1875] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:25.922293 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.946256 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:25.968315 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.969615 26107 ts_tablet_manager.cc:1888] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240503 08:15:25.969813 26107 log.cc:1196] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc9575f64eb741948e51ccf06f48228a
I20240503 08:15:25.998026 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:25.999686 14336 catalog_manager.cc:4830] TS c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): tablet dc9575f64eb741948e51ccf06f48228a (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:15:26.041067 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:26.054986 14619 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:26.076228 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:26.086742 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:26.087635 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:26.099589 16668 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
W20240503 08:15:26.101634 14555 consensus_peers.cc:485] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): Couldn't send request to peer b35c4c5d528c4a8f9e754ff9481c3938. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7a5221747794a13aa4ed763006ba291. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:26.102921 16405 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:26.103168 16667 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:26.104090 25700 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:26.105011 25700 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:26.163125 16404 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:15:26.163004 26163 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:26.164027 25182 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:26.167016 16403 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:26.168594 16667 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:26.174847 14348 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 7 to 8, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:26.210956 26172 ts_tablet_manager.cc:902] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:26.272251 16404 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:26.272253 26177 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:26.280014 26172 tablet_copy_client.cc:308] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:26.280858 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet dc9575f64eb741948e51ccf06f48228a from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:33947)
I20240503 08:15:26.281034 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet dc9575f64eb741948e51ccf06f48228a from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:33947: session id = 0d77c4b1f12049dea71b97e86be2b288-dc9575f64eb741948e51ccf06f48228a
I20240503 08:15:26.283792 14901 tablet_copy_source_session.cc:205] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:26.284812 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 11 to 12, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:26.290350 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:26.293460 26172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9575f64eb741948e51ccf06f48228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:26.299268 16667 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
W20240503 08:15:26.302878 16605 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:26.303488 16404 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:15:26.304369 25182 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:26.308254 16404 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:26.314344 26177 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:26.315361 14344 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:26.319403 14335 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4e86f4f3a01048e69f553c6557d79b71 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:26.334640 26172 tablet_copy_client.cc:783] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:26.334915 26172 tablet_copy_client.cc:647] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:26.347580 26172 tablet_copy_client.cc:523] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:26.351454 26172 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:15:26.359740 26191 tablet_replica.cc:330] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:26.360236 26191 raft_consensus.cc:2229] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:26.360697 26191 raft_consensus.cc:2258] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:26.361918 26191 ts_tablet_manager.cc:1875] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:26.396642 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:26.412678 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:26.438706 26191 ts_tablet_manager.cc:1888] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 08:15:26.442312 26191 log.cc:1196] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:26.443584 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet 4e86f4f3a01048e69f553c6557d79b71 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:26.455778 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:26.473882 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:26.537686 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:26.564019 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:26.576171 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:26.598943 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:26.620517 26172 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:26.620962 26172 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:26.621227 26172 ts_tablet_manager.cc:1367] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.270s	user 0.061s	sys 0.000s
I20240503 08:15:26.621961 26172 raft_consensus.cc:348] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:26.622177 26172 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:26.622386 26172 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:26.645222 26172 ts_tablet_manager.cc:1398] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.000s
I20240503 08:15:26.650532 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-dc9575f64eb741948e51ccf06f48228a received from {username='slave'} at 127.2.49.130:33947
I20240503 08:15:26.650696 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-dc9575f64eb741948e51ccf06f48228a on tablet dc9575f64eb741948e51ccf06f48228a with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:26.702695 26251 ts_tablet_manager.cc:902] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Initiating tablet copy from peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:15:26.733641 26256 ts_tablet_manager.cc:902] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:15:26.745186 26256 tablet_copy_client.cc:308] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:15:26.746147 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet f7a5221747794a13aa4ed763006ba291 from peer b35c4c5d528c4a8f9e754ff9481c3938 ({username='slave'} at 127.2.49.132:38085)
I20240503 08:15:26.746346 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet f7a5221747794a13aa4ed763006ba291 from peer b35c4c5d528c4a8f9e754ff9481c3938 at {username='slave'} at 127.2.49.132:38085: session id = b35c4c5d528c4a8f9e754ff9481c3938-f7a5221747794a13aa4ed763006ba291
I20240503 08:15:26.749212 14641 tablet_copy_source_session.cc:205] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:26.751161 26256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a5221747794a13aa4ed763006ba291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:26.779487 14619 raft_consensus.cc:1203] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9]   Dedup: 3.9->[]
I20240503 08:15:26.787245 26251 tablet_copy_client.cc:308] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.135:46299
I20240503 08:15:26.788066 16689 tablet_copy_service.cc:140] P 6835ad958e094ce4927135b942a2e8a8: Received BeginTabletCopySession request for tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 129a2ac87d7f416db6d47d79bb1c2b5f ({username='slave'} at 127.2.49.129:58937)
I20240503 08:15:26.788245 16689 tablet_copy_service.cc:161] P 6835ad958e094ce4927135b942a2e8a8: Beginning new tablet copy session on tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 129a2ac87d7f416db6d47d79bb1c2b5f at {username='slave'} at 127.2.49.129:58937: session id = 129a2ac87d7f416db6d47d79bb1c2b5f-4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:26.791266 16689 tablet_copy_source_session.cc:205] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:26.792634 26251 ts_tablet_manager.cc:1875] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:26.802105 26251 ts_tablet_manager.cc:1888] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240503 08:15:26.802959 26251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:26.806592 26251 tablet_copy_client.cc:783] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:26.806835 26251 tablet_copy_client.cc:647] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:26.808543 26251 tablet_copy_client.cc:523] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:26.806064 26256 tablet_copy_client.cc:783] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:26.812808 26256 tablet_copy_client.cc:647] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:26.815348 26251 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:15:26.826402 26256 tablet_copy_client.cc:523] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:26.862798 16801 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:26.877043 14749 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:26.878212 24869 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:26.886133 14880 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:26.886932 24869 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:26.890877 26256 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:15:26.914136 24869 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:26.914412 14749 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:26.924160 14880 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:26.927934 14348 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:26.968456 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet 169bd9ea6772420abf757dd55cf44d31 with delete_type TABLET_DATA_TOMBSTONED (TS 6835ad958e094ce4927135b942a2e8a8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54464
I20240503 08:15:26.993923 26301 tablet_replica.cc:330] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: stopping tablet replica
I20240503 08:15:26.994741 26301 raft_consensus.cc:2229] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:26.995147 26301 raft_consensus.cc:2258] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:26.996807 26301 ts_tablet_manager.cc:1875] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:27.092377 26301 ts_tablet_manager.cc:1888] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:15:27.123505 26301 log.cc:1196] T 169bd9ea6772420abf757dd55cf44d31 P 6835ad958e094ce4927135b942a2e8a8: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/wals/169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:27.126467 14335 catalog_manager.cc:4830] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): tablet 169bd9ea6772420abf757dd55cf44d31 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:27.135111 26251 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:27.141574 26251 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap complete.
I20240503 08:15:27.142069 26251 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.327s	user 0.077s	sys 0.025s
I20240503 08:15:27.155616 26251 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:27.156095 26251 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:15:27.158018 26251 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:27.158833 26251 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.000s
I20240503 08:15:27.161351 16689 tablet_copy_service.cc:342] P 6835ad958e094ce4927135b942a2e8a8: Request end of tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-4e86f4f3a01048e69f553c6557d79b71 received from {username='slave'} at 127.2.49.129:58937
I20240503 08:15:27.161516 16689 tablet_copy_service.cc:433] P 6835ad958e094ce4927135b942a2e8a8: ending tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-4e86f4f3a01048e69f553c6557d79b71 on tablet 4e86f4f3a01048e69f553c6557d79b71 with peer 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:15:27.180485 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.222920 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:27.247871 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.253060 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.280462 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:27.299014 14490 raft_consensus.cc:1203] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13]   Dedup: 3.13->[]
I20240503 08:15:27.329627 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:27.365633 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:27.410037 26256 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:27.410579 26256 tablet_bootstrap.cc:492] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap complete.
I20240503 08:15:27.410878 26256 ts_tablet_manager.cc:1367] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.520s	user 0.064s	sys 0.000s
I20240503 08:15:27.411813 26256 raft_consensus.cc:348] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:27.412082 26256 raft_consensus.cc:729] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:15:27.412351 26256 consensus_queue.cc:260] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:27.414494 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:27.412952 26256 ts_tablet_manager.cc:1398] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:15:27.426800 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-f7a5221747794a13aa4ed763006ba291 received from {username='slave'} at 127.2.49.132:38085
I20240503 08:15:27.426963 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-f7a5221747794a13aa4ed763006ba291 on tablet f7a5221747794a13aa4ed763006ba291 with peer b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:15:27.533766 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.536072 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:27.556062 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:27.569662 16404 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:27.578975 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.582646 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:27.619709 14619 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 08:15:27.620469 16341 consensus_peers.cc:485] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:27.621136 22114 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:27.648583 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:27.677142 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:27.681475 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.689000 14880 raft_consensus.cc:1203] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:27.732694 16668 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:15:27.733649 22114 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:27.752940 22114 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:27.775965 16668 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:27.810187 14344 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 reported cstate change: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 2 leader_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:27.843993 14621 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:27.860321 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.879451 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.899843 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:27.903719 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:27.946995 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:27.968389 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:28.008412 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:28.022774 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:28.032922 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:28.037569 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:28.090451 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:28.111702 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:28.121582 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:28.150679 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:28.157578 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:28.172829 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:28.234314 14880 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
W20240503 08:15:28.249929 14817 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:28.250835 16535 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:28.251405 26371 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:28.251631 16404 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:28.252609 26371 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:28.262233 26471 ts_tablet_manager.cc:902] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Initiating tablet copy from peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:15:28.308669 26483 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.309952 16403 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.316388 14344 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.319094 14490 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:28.322472 16535 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.333637 16403 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:15:28.335728 26455 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:15:28.352113 26296 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:28.353008 16403 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:28.358731 14348 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 5 to 7, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:28.362421 26471 tablet_copy_client.cc:308] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.133:38023
I20240503 08:15:28.363662 16425 tablet_copy_service.cc:140] P 0f8702fad8ea4d22840dab11bb27f7f2: Received BeginTabletCopySession request for tablet f65ab6dba1124573962e2da58aa1859c from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 ({username='slave'} at 127.2.49.131:42843)
I20240503 08:15:28.363858 16425 tablet_copy_service.cc:161] P 0f8702fad8ea4d22840dab11bb27f7f2: Beginning new tablet copy session on tablet f65ab6dba1124573962e2da58aa1859c from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 at {username='slave'} at 127.2.49.131:42843: session id = c0fd1ebec9da48d4b0c1ae0f0ee96452-f65ab6dba1124573962e2da58aa1859c
I20240503 08:15:28.369961 16425 tablet_copy_source_session.cc:205] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:28.372592 14599 tablet_service.cc:1514] Processing DeleteTablet for tablet dd11460051ab40febc2fe6aff58b5028 with delete_type TABLET_DATA_TOMBSTONED (TS 0d77c4b1f12049dea71b97e86be2b288 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39240
I20240503 08:15:28.375867 14490 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:28.378077 16403 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 08:15:28.379439 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd11460051ab40febc2fe6aff58b5028. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:28.379757 26455 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:28.383162 26455 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:28.387495 16403 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:28.395576 26471 ts_tablet_manager.cc:1875] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:28.409765 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dd11460051ab40febc2fe6aff58b5028 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:15:28.413426 14344 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 7 to 8, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:15:28.415056 26471 ts_tablet_manager.cc:1888] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:15:28.416005 26471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65ab6dba1124573962e2da58aa1859c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:28.422664 26495 tablet_replica.cc:330] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: stopping tablet replica
I20240503 08:15:28.423203 26495 raft_consensus.cc:2229] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:28.423409 26471 tablet_copy_client.cc:783] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:28.423738 26471 tablet_copy_client.cc:647] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:28.424027 26495 raft_consensus.cc:2258] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:28.425211 26495 ts_tablet_manager.cc:1875] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:28.427537 26471 tablet_copy_client.cc:523] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:28.437696 26471 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:15:28.466071 26495 ts_tablet_manager.cc:1888] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:15:28.467242 26495 log.cc:1196] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd11460051ab40febc2fe6aff58b5028
I20240503 08:15:28.468626 14333 catalog_manager.cc:4830] TS 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): tablet dd11460051ab40febc2fe6aff58b5028 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:28.544149 26471 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:28.544741 26471 tablet_bootstrap.cc:492] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap complete.
I20240503 08:15:28.545058 26471 ts_tablet_manager.cc:1367] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.107s	user 0.052s	sys 0.003s
I20240503 08:15:28.545980 26471 raft_consensus.cc:348] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:28.546286 26471 raft_consensus.cc:729] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:15:28.546607 26471 consensus_queue.cc:260] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:28.547502 26471 ts_tablet_manager.cc:1398] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.002s
I20240503 08:15:28.548408 16425 tablet_copy_service.cc:342] P 0f8702fad8ea4d22840dab11bb27f7f2: Request end of tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-f65ab6dba1124573962e2da58aa1859c received from {username='slave'} at 127.2.49.131:42843
I20240503 08:15:28.548576 16425 tablet_copy_service.cc:433] P 0f8702fad8ea4d22840dab11bb27f7f2: ending tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-f65ab6dba1124573962e2da58aa1859c on tablet f65ab6dba1124573962e2da58aa1859c with peer c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:15:28.677184 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:28.706081 14878 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:28.709640 16667 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
W20240503 08:15:28.709875 14817 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:28.711306 14619 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 08:15:28.711732 26371 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:28.713198 26483 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:28.718360 26483 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:28.718796 14619 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:28.726797 14348 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 9 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:28.735148 14749 raft_consensus.cc:1203] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 08:15:28.739620 26561 ts_tablet_manager.cc:902] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:28.744221 16667 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
W20240503 08:15:28.753373 14817 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:28.755028 16403 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:15:28.755669 26567 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:28.758634 16801 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:15:28.762715 26371 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:28.771627 26567 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.772411 16801 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.782106 14344 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.789579 16403 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.826740 14749 consensus_queue.cc:237] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:28.840040 14490 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:28.841809 26190 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:28.850008 16801 raft_consensus.cc:1261] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:28.851465 26190 consensus_queue.cc:1035] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:28.908144 26601 ts_tablet_manager.cc:902] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:28.926760 26617 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.928784 14490 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.945771 14348 catalog_manager.cc:5484] T 897a701b880641b08816f2d1a254ef03 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:28.954438 16516 tablet_service.cc:1514] Processing DeleteTablet for tablet 897a701b880641b08816f2d1a254ef03 with delete_type TABLET_DATA_TOMBSTONED (TS 95860e22c9474e2d894f04deba8442cc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43672
I20240503 08:15:28.965700 26561 tablet_copy_client.cc:308] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
W20240503 08:15:28.966418 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 57720us
I20240503 08:15:28.966866 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 5d16c2981213464cbefa44b6ca509182 from peer 129a2ac87d7f416db6d47d79bb1c2b5f ({username='slave'} at 127.2.49.129:53921)
I20240503 08:15:28.967125 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 5d16c2981213464cbefa44b6ca509182 from peer 129a2ac87d7f416db6d47d79bb1c2b5f at {username='slave'} at 127.2.49.129:53921: session id = 129a2ac87d7f416db6d47d79bb1c2b5f-5d16c2981213464cbefa44b6ca509182
I20240503 08:15:28.971027 14901 tablet_copy_source_session.cc:205] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:28.977322 26561 ts_tablet_manager.cc:1875] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 08:15:28.980243 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 98038.8us
I20240503 08:15:28.994410 26623 tablet_replica.cc:330] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: stopping tablet replica
I20240503 08:15:28.994805 26623 raft_consensus.cc:2229] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:28.995605 26623 raft_consensus.cc:2258] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:28.996711 26623 ts_tablet_manager.cc:1875] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:29.029891 26623 ts_tablet_manager.cc:1888] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:29.030210 26623 log.cc:1196] T 897a701b880641b08816f2d1a254ef03 P 95860e22c9474e2d894f04deba8442cc: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/wals/897a701b880641b08816f2d1a254ef03
I20240503 08:15:29.030194 26601 tablet_copy_client.cc:308] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:29.031216 14336 catalog_manager.cc:4830] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): tablet 897a701b880641b08816f2d1a254ef03 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:15:29.031253 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet dd11460051ab40febc2fe6aff58b5028 from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:55303)
I20240503 08:15:29.031448 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet dd11460051ab40febc2fe6aff58b5028 from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:55303: session id = 036223b0890d4634acfadf9596f39f80-dd11460051ab40febc2fe6aff58b5028
I20240503 08:15:29.035737 14507 tablet_copy_source_session.cc:205] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:29.035737 26561 ts_tablet_manager.cc:1888] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:29.036700 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d16c2981213464cbefa44b6ca509182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:29.038002 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd11460051ab40febc2fe6aff58b5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:29.043928 26561 tablet_copy_client.cc:783] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:29.044354 26561 tablet_copy_client.cc:647] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:29.048604 26561 tablet_copy_client.cc:523] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:29.054055 26561 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:15:29.055534 16801 raft_consensus.cc:2932] T 897a701b880641b08816f2d1a254ef03 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:29.086051 26601 tablet_copy_client.cc:783] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:29.086398 26601 tablet_copy_client.cc:647] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:29.105083 26601 tablet_copy_client.cc:523] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:29.109299 26601 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
I20240503 08:15:29.231839 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:29.268371 26681 ts_tablet_manager.cc:902] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:29.272130 26601 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:29.272658 26601 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:15:29.272962 26601 ts_tablet_manager.cc:1367] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.164s	user 0.058s	sys 0.009s
I20240503 08:15:29.273828 26601 raft_consensus.cc:348] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:29.274061 26601 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:15:29.274333 26601 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:29.274895 26601 ts_tablet_manager.cc:1398] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:15:29.279508 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:29.281261 26693 ts_tablet_manager.cc:902] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:29.282663 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-dd11460051ab40febc2fe6aff58b5028 received from {username='slave'} at 127.2.49.136:55303
I20240503 08:15:29.282799 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 036223b0890d4634acfadf9596f39f80-dd11460051ab40febc2fe6aff58b5028 on tablet dd11460051ab40febc2fe6aff58b5028 with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:15:29.337021 16667 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 8, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:29.343281 14880 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:29.363143 16403 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
W20240503 08:15:29.363380 16602 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:29.364044 26177 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:29.364775 26681 tablet_copy_client.cc:308] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:29.366119 14490 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 08:15:29.367573 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet dc9575f64eb741948e51ccf06f48228a from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 ({username='slave'} at 127.2.49.131:56291)
I20240503 08:15:29.367534 26710 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:29.367748 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet dc9575f64eb741948e51ccf06f48228a from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 at {username='slave'} at 127.2.49.131:56291: session id = c0fd1ebec9da48d4b0c1ae0f0ee96452-dc9575f64eb741948e51ccf06f48228a
I20240503 08:15:29.370662 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:29.373903 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:29.374749 14901 tablet_copy_source_session.cc:205] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:29.376317 26681 ts_tablet_manager.cc:1875] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 08:15:29.389338 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:29.390492 16535 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:29.391146 26567 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:29.393605 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:29.402145 16667 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:29.402810 26567 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:29.416023 26710 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:29.416579 14490 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:29.425482 26681 ts_tablet_manager.cc:1888] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.7
I20240503 08:15:29.425787 26681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9575f64eb741948e51ccf06f48228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:29.435020 14348 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 13 to 14, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:29.456741 16404 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:29.440894 26681 tablet_copy_client.cc:783] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:29.463665 26681 tablet_copy_client.cc:647] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:29.467402 26681 tablet_copy_client.cc:523] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:29.470643 16801 raft_consensus.cc:1203] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 08:15:29.495568 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:29.499073 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:29.518733 26731 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:29.524906 26681 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:15:29.529911 26693 tablet_copy_client.cc:308] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:29.530855 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 72a949e76ab5477680a4f9912e9672e9 from peer 129a2ac87d7f416db6d47d79bb1c2b5f ({username='slave'} at 127.2.49.129:53921)
I20240503 08:15:29.531066 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 72a949e76ab5477680a4f9912e9672e9 from peer 129a2ac87d7f416db6d47d79bb1c2b5f at {username='slave'} at 127.2.49.129:53921: session id = 129a2ac87d7f416db6d47d79bb1c2b5f-72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:29.534072 14901 tablet_copy_source_session.cc:205] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:29.535508 26693 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:29.535060 16535 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:29.562626 26693 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.10
I20240503 08:15:29.568820 26693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:29.574632 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:29.580027 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:29.591094 26693 tablet_copy_client.cc:783] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:29.591414 26693 tablet_copy_client.cc:647] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:29.593676 26693 tablet_copy_client.cc:523] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:29.606848 14348 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc reported cstate change: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:29.610126 26561 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:29.610608 26561 tablet_bootstrap.cc:492] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap complete.
I20240503 08:15:29.610872 26561 ts_tablet_manager.cc:1367] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.557s	user 0.044s	sys 0.023s
I20240503 08:15:29.611585 26561 raft_consensus.cc:348] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:29.611800 26561 raft_consensus.cc:729] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:15:29.612059 26561 consensus_queue.cc:260] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:29.612653 26561 ts_tablet_manager.cc:1398] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:15:29.613560 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-5d16c2981213464cbefa44b6ca509182 received from {username='slave'} at 127.2.49.129:53921
I20240503 08:15:29.613723 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-5d16c2981213464cbefa44b6ca509182 on tablet 5d16c2981213464cbefa44b6ca509182 with peer 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:15:29.626716 16667 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:29.631778 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:29.656024 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:29.681366 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:29.696103 14490 raft_consensus.cc:1203] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:29.696292 26693 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:15:29.705065 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:29.712807 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:29.750921 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:29.764024 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:29.829363 26768 ts_tablet_manager.cc:902] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:29.887950 26768 tablet_copy_client.cc:308] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:29.889123 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 55067e08fb694e90b6366d41cc962095 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 ({username='slave'} at 127.2.49.131:56291)
I20240503 08:15:29.889302 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 55067e08fb694e90b6366d41cc962095 from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 at {username='slave'} at 127.2.49.131:56291: session id = c0fd1ebec9da48d4b0c1ae0f0ee96452-55067e08fb694e90b6366d41cc962095
I20240503 08:15:29.893078 14901 tablet_copy_source_session.cc:205] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:29.897814 26768 ts_tablet_manager.cc:1875] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:29.918732 26776 ts_tablet_manager.cc:902] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299)
I20240503 08:15:29.919564 26768 ts_tablet_manager.cc:1888] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:29.919870 26768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:29.949070 26776 tablet_copy_client.cc:308] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.135:46299
I20240503 08:15:29.949982 16689 tablet_copy_service.cc:140] P 6835ad958e094ce4927135b942a2e8a8: Received BeginTabletCopySession request for tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:51523)
I20240503 08:15:29.950182 16689 tablet_copy_service.cc:161] P 6835ad958e094ce4927135b942a2e8a8: Beginning new tablet copy session on tablet 4e86f4f3a01048e69f553c6557d79b71 from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:51523: session id = 0d77c4b1f12049dea71b97e86be2b288-4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:29.953038 16689 tablet_copy_source_session.cc:205] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:29.954295 26776 ts_tablet_manager.cc:1875] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:29.967756 26768 tablet_copy_client.cc:783] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:29.968093 26768 tablet_copy_client.cc:647] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:29.968720 26776 ts_tablet_manager.cc:1888] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.11
I20240503 08:15:29.971879 26776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86f4f3a01048e69f553c6557d79b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:29.976286 26768 tablet_copy_client.cc:523] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:29.979750 26768 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap starting.
I20240503 08:15:29.982270 26776 tablet_copy_client.cc:783] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:29.982561 26776 tablet_copy_client.cc:647] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:29.986456 26776 tablet_copy_client.cc:523] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:29.998574 26776 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:15:30.047943 26681 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:30.053649 26681 tablet_bootstrap.cc:492] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap complete.
I20240503 08:15:30.053952 26681 ts_tablet_manager.cc:1367] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.529s	user 0.052s	sys 0.010s
I20240503 08:15:30.054643 26681 raft_consensus.cc:348] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:30.054872 26681 raft_consensus.cc:729] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:15:30.055598 26693 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:30.058851 26693 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap complete.
I20240503 08:15:30.059204 26693 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.363s	user 0.052s	sys 0.011s
I20240503 08:15:30.062276 26693 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:30.063539 26693 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:15:30.063886 26693 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:30.065723 26693 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 08:15:30.069067 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-72a949e76ab5477680a4f9912e9672e9 received from {username='slave'} at 127.2.49.129:53921
I20240503 08:15:30.069193 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-72a949e76ab5477680a4f9912e9672e9 on tablet 72a949e76ab5477680a4f9912e9672e9 with peer 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:15:30.074904 26681 consensus_queue.cc:260] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:30.075625 26681 ts_tablet_manager.cc:1398] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.022s	user 0.002s	sys 0.000s
I20240503 08:15:30.076571 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-dc9575f64eb741948e51ccf06f48228a received from {username='slave'} at 127.2.49.131:56291
I20240503 08:15:30.076705 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-dc9575f64eb741948e51ccf06f48228a on tablet dc9575f64eb741948e51ccf06f48228a with peer c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:15:30.104877 14749 raft_consensus.cc:1203] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.9->[3.10-3.10]   Dedup: 3.10->[]
I20240503 08:15:30.144037 26776 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:30.144511 26776 tablet_bootstrap.cc:492] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:30.144856 26776 ts_tablet_manager.cc:1367] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.146s	user 0.067s	sys 0.000s
I20240503 08:15:30.145640 26776 raft_consensus.cc:348] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:30.145879 26776 raft_consensus.cc:729] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:30.146194 26776 consensus_queue.cc:260] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:30.146946 26776 ts_tablet_manager.cc:1398] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:15:30.151085 16689 tablet_copy_service.cc:342] P 6835ad958e094ce4927135b942a2e8a8: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-4e86f4f3a01048e69f553c6557d79b71 received from {username='slave'} at 127.2.49.130:51523
I20240503 08:15:30.151263 16689 tablet_copy_service.cc:433] P 6835ad958e094ce4927135b942a2e8a8: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-4e86f4f3a01048e69f553c6557d79b71 on tablet 4e86f4f3a01048e69f553c6557d79b71 with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:30.175272 26768 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:30.175681 26768 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Bootstrap complete.
I20240503 08:15:30.175948 26768 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent bootstrapping tablet: real 0.196s	user 0.058s	sys 0.006s
I20240503 08:15:30.176813 26768 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:30.177042 26768 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd1ebec9da48d4b0c1ae0f0ee96452, State: Initialized, Role: FOLLOWER
I20240503 08:15:30.177345 26768 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:30.177963 26768 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:15:30.179162 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-55067e08fb694e90b6366d41cc962095 received from {username='slave'} at 127.2.49.131:56291
I20240503 08:15:30.179296 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session c0fd1ebec9da48d4b0c1ae0f0ee96452-55067e08fb694e90b6366d41cc962095 on tablet 55067e08fb694e90b6366d41cc962095 with peer c0fd1ebec9da48d4b0c1ae0f0ee96452
I20240503 08:15:30.238999 16801 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:30.245195 14880 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:15:30.246127 14749 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 036223b0890d4634acfadf9596f39f80: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:15:30.246121 24869 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:15:30.247315 16734 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:30.249404 24869 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:30.252368 26767 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.257740 14749 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.270491 14348 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 10 to 11, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 2 leader_uuid: "036223b0890d4634acfadf9596f39f80" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.289451 14880 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.310921 14749 raft_consensus.cc:1203] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:30.324079 14490 raft_consensus.cc:1203] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.12->[2.13-2.13]   Dedup: 2.13->[]
I20240503 08:15:30.374553 14619 raft_consensus.cc:1203] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.13->[3.14-3.14]   Dedup: 3.14->[]
I20240503 08:15:30.439760 14490 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
W20240503 08:15:30.451051 14423 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:30.451284 16404 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:15:30.451910 26455 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:30.457134 16801 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:15:30.457918 26884 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:30.467330 26884 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.470933 16800 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.473307 16404 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.482100 14340 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 8 to 9, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:30.700371 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.703492 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:30.720995 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:30.740612 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:30.745982 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:30.746538 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:30.764029 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.768926 16780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.768993 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.787438 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.798228 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:30.830094 26955 ts_tablet_manager.cc:902] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:15:30.836472 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.842063 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:30.844198 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.884864 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:30.920686 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:30.930299 26958 ts_tablet_manager.cc:902] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:30.955143 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.963723 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:30.963832 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:30.984498 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:31.007628 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:31.032737 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:31.045195 26958 tablet_copy_client.cc:308] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:31.047212 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet dd11460051ab40febc2fe6aff58b5028 from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:47845)
I20240503 08:15:31.047369 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet dd11460051ab40febc2fe6aff58b5028 from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:47845: session id = 0d77c4b1f12049dea71b97e86be2b288-dd11460051ab40febc2fe6aff58b5028
I20240503 08:15:31.052124 14507 tablet_copy_source_session.cc:205] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:31.054349 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:31.067898 14749 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
W20240503 08:15:31.087170 14684 consensus_peers.cc:485] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c130eb7ea8a74f4b8d9b79e23af70067. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:31.089458 16404 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:31.094867 26619 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:31.102499 26955 tablet_copy_client.cc:308] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.136:37641
I20240503 08:15:31.103204 16800 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:15:31.103446 16821 tablet_copy_service.cc:140] P 036223b0890d4634acfadf9596f39f80: Received BeginTabletCopySession request for tablet 169bd9ea6772420abf757dd55cf44d31 from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:39317)
I20240503 08:15:31.103606 16821 tablet_copy_service.cc:161] P 036223b0890d4634acfadf9596f39f80: Beginning new tablet copy session on tablet 169bd9ea6772420abf757dd55cf44d31 from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:39317: session id = 0d77c4b1f12049dea71b97e86be2b288-169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:31.104068 26619 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:15:31.106631 16821 tablet_copy_source_session.cc:205] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:31.107846 26955 ts_tablet_manager.cc:1875] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:31.086537 26958 ts_tablet_manager.cc:1875] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:31.118070 26955 ts_tablet_manager.cc:1888] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240503 08:15:31.119218 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169bd9ea6772420abf757dd55cf44d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:31.143114 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:31.154186 26985 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.155126 16404 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.160323 16800 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.167244 14344 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 7 to 8, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.169415 26955 tablet_copy_client.cc:783] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:31.193284 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.196357 26955 tablet_copy_client.cc:647] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:31.218237 26955 tablet_copy_client.cc:523] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:15:31.219810 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 44584.5us
I20240503 08:15:31.235430 26955 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
W20240503 08:15:31.253932 14423 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:31.255558 16536 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:31.256376 26455 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:31.261814 16668 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:31.262929 26455 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:31.265897 26958 ts_tablet_manager.cc:1888] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 08:15:31.266291 26958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd11460051ab40febc2fe6aff58b5028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:31.269274 26958 tablet_copy_client.cc:783] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:31.269481 26958 tablet_copy_client.cc:647] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:31.282752 26455 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.289095 16667 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.291728 26958 tablet_copy_client.cc:523] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:31.309891 16536 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.317495 26958 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:15:31.325517 14340 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 9 to 10, VOTER 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.407902 26958 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:31.408399 26958 tablet_bootstrap.cc:492] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:31.408672 26958 ts_tablet_manager.cc:1367] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.091s	user 0.043s	sys 0.012s
I20240503 08:15:31.409449 26958 raft_consensus.cc:348] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.409698 26958 raft_consensus.cc:729] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:31.409957 26958 consensus_queue.cc:260] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.410512 26958 ts_tablet_manager.cc:1398] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:15:31.413594 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-dd11460051ab40febc2fe6aff58b5028 received from {username='slave'} at 127.2.49.130:47845
I20240503 08:15:31.413712 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-dd11460051ab40febc2fe6aff58b5028 on tablet dd11460051ab40febc2fe6aff58b5028 with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:31.420753 14619 raft_consensus.cc:1203] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 08:15:31.456065 26955 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:31.456507 26955 tablet_bootstrap.cc:492] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:31.456748 26955 ts_tablet_manager.cc:1367] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.221s	user 0.054s	sys 0.004s
I20240503 08:15:31.457567 26955 raft_consensus.cc:348] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.457847 26955 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:31.458101 26955 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.459363 26955 ts_tablet_manager.cc:1398] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:15:31.460178 16821 tablet_copy_service.cc:342] P 036223b0890d4634acfadf9596f39f80: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-169bd9ea6772420abf757dd55cf44d31 received from {username='slave'} at 127.2.49.130:39317
I20240503 08:15:31.460297 16821 tablet_copy_service.cc:433] P 036223b0890d4634acfadf9596f39f80: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-169bd9ea6772420abf757dd55cf44d31 on tablet 169bd9ea6772420abf757dd55cf44d31 with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:31.607601 26955 ts_tablet_manager.cc:902] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:15:31.636997 26955 tablet_copy_client.cc:308] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:45663
I20240503 08:15:31.637881 14771 tablet_copy_service.cc:140] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Received BeginTabletCopySession request for tablet c130eb7ea8a74f4b8d9b79e23af70067 from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:41213)
I20240503 08:15:31.638068 14771 tablet_copy_service.cc:161] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Beginning new tablet copy session on tablet c130eb7ea8a74f4b8d9b79e23af70067 from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:41213: session id = 0d77c4b1f12049dea71b97e86be2b288-c130eb7ea8a74f4b8d9b79e23af70067
I20240503 08:15:31.641431 14771 tablet_copy_source_session.cc:205] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:31.644970 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c130eb7ea8a74f4b8d9b79e23af70067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:31.664264 26955 tablet_copy_client.cc:783] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:31.664534 26955 tablet_copy_client.cc:647] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:31.667357 26958 ts_tablet_manager.cc:902] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:31.678339 26955 tablet_copy_client.cc:523] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:31.685250 26955 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:15:31.750375 26958 tablet_copy_client.cc:308] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:15:31.751775 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer 0d77c4b1f12049dea71b97e86be2b288 ({username='slave'} at 127.2.49.130:47845)
I20240503 08:15:31.751960 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet da1f165b4a5647a48ad7ff951fb3fbf6 from peer 0d77c4b1f12049dea71b97e86be2b288 at {username='slave'} at 127.2.49.130:47845: session id = 0d77c4b1f12049dea71b97e86be2b288-da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:31.755036 14507 tablet_copy_source_session.cc:205] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:31.758792 26958 ts_tablet_manager.cc:1875] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:31.767400 16667 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.768251 26958 ts_tablet_manager.cc:1888] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:15:31.768525 26958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f165b4a5647a48ad7ff951fb3fbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:31.775707 14880 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:31.779044 26955 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:31.779651 26955 tablet_bootstrap.cc:492] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:31.779979 26955 ts_tablet_manager.cc:1367] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.095s	user 0.053s	sys 0.004s
I20240503 08:15:31.781071 14619 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 08:15:31.782212 26723 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:31.783908 26958 tablet_copy_client.cc:783] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:31.784132 26958 tablet_copy_client.cc:647] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:31.787756 14490 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Refusing update from remote peer 6835ad958e094ce4927135b942a2e8a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 08:15:31.788750 26723 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:31.781075 26955 raft_consensus.cc:348] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.790333 26955 raft_consensus.cc:729] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:31.792189 26955 consensus_queue.cc:260] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:31.795328 26955 ts_tablet_manager.cc:1398] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.000s
I20240503 08:15:31.795733 26958 tablet_copy_client.cc:523] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:31.797132 14771 tablet_copy_service.cc:342] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-c130eb7ea8a74f4b8d9b79e23af70067 received from {username='slave'} at 127.2.49.130:41213
I20240503 08:15:31.797300 14771 tablet_copy_service.cc:433] P c0fd1ebec9da48d4b0c1ae0f0ee96452: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-c130eb7ea8a74f4b8d9b79e23af70067 on tablet c130eb7ea8a74f4b8d9b79e23af70067 with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:31.815848 26958 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap starting.
I20240503 08:15:31.816622 14749 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:31.823958 26731 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:31.828917 16536 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:31.829046 27092 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 6835ad958e094ce4927135b942a2e8a8 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.830121 27040 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:31.830770 14490 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.834589 14619 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.835928 14621 raft_consensus.cc:1203] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11]   Dedup: 2.11->[]
I20240503 08:15:31.840833 14340 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New cstate: current_term: 3 leader_uuid: "6835ad958e094ce4927135b942a2e8a8" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:31.849298 26731 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:31.851778 16535 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:31.904470 14348 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:31.906742 14748 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:31.913080 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86f4f3a01048e69f553c6557d79b71 with delete_type TABLET_DATA_TOMBSTONED (TS 0f8702fad8ea4d22840dab11bb27f7f2 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:43360
I20240503 08:15:31.927461 27109 tablet_replica.cc:330] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: stopping tablet replica
I20240503 08:15:31.928349 27109 raft_consensus.cc:2229] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:31.930220 27109 raft_consensus.cc:2258] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:31.931599 27109 ts_tablet_manager.cc:1875] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:31.932121 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet 55067e08fb694e90b6366d41cc962095 with delete_type TABLET_DATA_TOMBSTONED (TS 6835ad958e094ce4927135b942a2e8a8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54464
I20240503 08:15:31.941532 27109 ts_tablet_manager.cc:1888] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240503 08:15:31.945565 27109 log.cc:1196] T 4e86f4f3a01048e69f553c6557d79b71 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/wals/4e86f4f3a01048e69f553c6557d79b71
I20240503 08:15:31.947099 14334 catalog_manager.cc:4830] TS 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): tablet 4e86f4f3a01048e69f553c6557d79b71 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:31.962396 27117 tablet_replica.cc:330] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: stopping tablet replica
I20240503 08:15:31.962934 27117 raft_consensus.cc:2229] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:31.963285 27117 raft_consensus.cc:2258] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:31.965481 27117 ts_tablet_manager.cc:1875] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:32.004604 27117 ts_tablet_manager.cc:1888] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:32.017160 27117 log.cc:1196] T 55067e08fb694e90b6366d41cc962095 P 6835ad958e094ce4927135b942a2e8a8: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/wals/55067e08fb694e90b6366d41cc962095
I20240503 08:15:32.031411 14335 catalog_manager.cc:4830] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): tablet 55067e08fb694e90b6366d41cc962095 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:32.106660 26958 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:32.107096 26958 tablet_bootstrap.cc:492] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Bootstrap complete.
I20240503 08:15:32.107391 26958 ts_tablet_manager.cc:1367] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Time spent bootstrapping tablet: real 0.292s	user 0.088s	sys 0.013s
I20240503 08:15:32.108100 26958 raft_consensus.cc:348] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:32.108311 26958 raft_consensus.cc:729] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Initialized, Role: FOLLOWER
I20240503 08:15:32.108515 26958 consensus_queue.cc:260] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:32.110452 26958 ts_tablet_manager.cc:1398] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:15:32.111315 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session 0d77c4b1f12049dea71b97e86be2b288-da1f165b4a5647a48ad7ff951fb3fbf6 received from {username='slave'} at 127.2.49.130:47845
I20240503 08:15:32.111455 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session 0d77c4b1f12049dea71b97e86be2b288-da1f165b4a5647a48ad7ff951fb3fbf6 on tablet da1f165b4a5647a48ad7ff951fb3fbf6 with peer 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:32.114907 14619 raft_consensus.cc:1203] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:15:32.172263 14619 raft_consensus.cc:1203] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240503 08:15:32.275668 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.287180 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.333842 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:32.336205 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:32.344980 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.352006 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:15:32.380070 16737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:45663, user_credentials={real_user=slave}} blocked reactor thread for 44230us
I20240503 08:15:32.415212 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.422591 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.426841 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:32.436798 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:32.449770 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:32.459216 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:32.475464 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.512194 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.513648 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:32.518520 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:32.560866 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:32.561200 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:32.563617 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.627383 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:32.679724 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:32.687209 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:32.691274 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:32.752158 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:33.225996 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:33.258565 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:33.274901 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:33.302772 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:33.316128 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:33.354511 14748 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:33.369074 16800 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:33.370203 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:33.370558 26986 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:33.375959 14619 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:33.376917 26986 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:33.399230 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:33.411443 26986 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.416904 16801 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.417963 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:33.425482 14344 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.436451 14621 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.466970 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:33.496783 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:33.504468 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:33.525573 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:33.565587 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:33.565726 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:33.570427 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:33.625883 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:33.637444 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet c130eb7ea8a74f4b8d9b79e23af70067 with delete_type TABLET_DATA_TOMBSTONED (TS 0f8702fad8ea4d22840dab11bb27f7f2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43360
I20240503 08:15:33.666755 14489 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:33.672621 14621 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:33.672641 16536 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:33.673205 26455 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:33.674072 27259 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:33.680259 27317 tablet_replica.cc:330] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: stopping tablet replica
I20240503 08:15:33.680501 27317 raft_consensus.cc:2229] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:33.680860 27317 raft_consensus.cc:2258] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:33.688514 27317 ts_tablet_manager.cc:1875] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:33.700389 27317 ts_tablet_manager.cc:1888] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:33.700541 27317 log.cc:1196] T c130eb7ea8a74f4b8d9b79e23af70067 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/wals/c130eb7ea8a74f4b8d9b79e23af70067
I20240503 08:15:33.701306 27259 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.701572 14334 catalog_manager.cc:4830] TS 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): tablet c130eb7ea8a74f4b8d9b79e23af70067 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:33.705200 14619 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.713325 14344 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.717861 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet da1f165b4a5647a48ad7ff951fb3fbf6 with delete_type TABLET_DATA_TOMBSTONED (TS 6835ad958e094ce4927135b942a2e8a8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54464
I20240503 08:15:33.733843 27324 tablet_replica.cc:330] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: stopping tablet replica
I20240503 08:15:33.734478 27324 raft_consensus.cc:2229] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:33.734944 27324 raft_consensus.cc:2258] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:33.740447 16536 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:33.747196 27324 ts_tablet_manager.cc:1875] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:33.755749 27324 ts_tablet_manager.cc:1888] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:15:33.758438 27324 log.cc:1196] T da1f165b4a5647a48ad7ff951fb3fbf6 P 6835ad958e094ce4927135b942a2e8a8: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/wals/da1f165b4a5647a48ad7ff951fb3fbf6
I20240503 08:15:33.763860 14335 catalog_manager.cc:4830] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): tablet da1f165b4a5647a48ad7ff951fb3fbf6 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:34.074410 14878 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:34.088805 16801 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:15:34.089869 27308 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:34.090205 14489 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:15:34.092478 27308 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:34.118049 27393 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:34.120267 14878 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:34.120621 16801 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:34.128090 14340 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 13 to 14, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:34.133031 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a949e76ab5477680a4f9912e9672e9 with delete_type TABLET_DATA_TOMBSTONED (TS 0f8702fad8ea4d22840dab11bb27f7f2 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:43360
I20240503 08:15:34.133399 27317 tablet_replica.cc:330] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: stopping tablet replica
I20240503 08:15:34.133042 14490 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:34.133621 27317 raft_consensus.cc:2229] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:34.133857 27317 raft_consensus.cc:2258] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:34.134619 27317 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:34.142280 27317 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 08:15:34.142555 27317 log.cc:1196] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/wals/72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:34.148458 14621 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:15:34.151032 14748 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:15:34.151011 27308 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:34.151796 14334 catalog_manager.cc:4830] TS 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): tablet 72a949e76ab5477680a4f9912e9672e9 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
I20240503 08:15:34.156113 27399 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:34.177147 27400 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:34.181572 14621 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:34.182444 14748 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:34.184307 14340 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:34.193046 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9575f64eb741948e51ccf06f48228a with delete_type TABLET_DATA_TOMBSTONED (TS 6835ad958e094ce4927135b942a2e8a8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54464
I20240503 08:15:34.193439 27324 tablet_replica.cc:330] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: stopping tablet replica
I20240503 08:15:34.193682 27324 raft_consensus.cc:2229] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:34.194005 27324 raft_consensus.cc:2258] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:34.197645 27324 ts_tablet_manager.cc:1875] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:34.208827 27324 ts_tablet_manager.cc:1888] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240503 08:15:34.209111 27324 log.cc:1196] T dc9575f64eb741948e51ccf06f48228a P 6835ad958e094ce4927135b942a2e8a8: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/wals/dc9575f64eb741948e51ccf06f48228a
I20240503 08:15:34.210129 14335 catalog_manager.cc:4830] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): tablet dc9575f64eb741948e51ccf06f48228a (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:15:34.498337 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:34.518079 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:34.531701 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:34.560622 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:34.583946 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:34.596483 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:34.613991 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:34.618968 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:34.632942 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:34.672458 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:34.696161 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:34.715087 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:34.715545 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:34.775224 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:34.787812 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:34.798856 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:35.118456 16781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:35.167470 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:35.175079 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:35.207558 16801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31"
dest_uuid: "036223b0890d4634acfadf9596f39f80"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20240503 08:15:35.207851 16801 raft_consensus.cc:595] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:15:35.212182 14860 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:15:35.249514 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:35.255429 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:35.256045 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:35.299337 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:35.407135 14490 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:35.435609 14621 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:35.436451 27482 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:35.449658 16404 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:35.450796 27482 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:35.468168 27482 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:35.472532 16403 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:35.477476 27574 raft_consensus.cc:979] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: : Instructing follower b35c4c5d528c4a8f9e754ff9481c3938 to start an election
I20240503 08:15:35.478413 26854 raft_consensus.cc:1067] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Signalling peer b35c4c5d528c4a8f9e754ff9481c3938 to start an election
I20240503 08:15:35.479728 14878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31"
dest_uuid: "b35c4c5d528c4a8f9e754ff9481c3938"
 from {username='slave'} at 127.2.49.136:41491
I20240503 08:15:35.479959 14878 raft_consensus.cc:482] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:15:35.480062 14878 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:35.481755 14349 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 9 to 10, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:35.488253 14878 raft_consensus.cc:504] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:35.490056 16801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "036223b0890d4634acfadf9596f39f80"
I20240503 08:15:35.491863 14748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:15:35.492134 14748 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:35.495510 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169bd9ea6772420abf757dd55cf44d31" candidate_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "0d77c4b1f12049dea71b97e86be2b288"
I20240503 08:15:35.495764 14619 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:35.498816 14878 leader_election.cc:288] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 3 election: Requested vote from peers 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:15:35.500998 14748 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 3.
I20240503 08:15:35.502485 14619 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 3.
I20240503 08:15:35.503043 14814 leader_election.cc:302] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, b35c4c5d528c4a8f9e754ff9481c3938, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 
I20240503 08:15:35.503597 27399 raft_consensus.cc:2781] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:15:35.508989 16801 raft_consensus.cc:3032] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:15:35.509145 16801 raft_consensus.cc:729] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Running, Role: LEADER
I20240503 08:15:35.509550 16801 consensus_queue.cc:260] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:35.510210 16801 raft_consensus.cc:3037] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:15:35.513675 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:35.514180 16801 raft_consensus.cc:2454] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c4c5d528c4a8f9e754ff9481c3938 in term 3.
I20240503 08:15:35.517181 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:35.529392 14619 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:35.548835 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:35.564378 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:35.592085 27399 raft_consensus.cc:686] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Becoming Leader. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Running, Role: LEADER
I20240503 08:15:35.592523 27399 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:35.597929 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:35.609061 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:35.624722 14340 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: term changed from 2 to 3, leader changed from 036223b0890d4634acfadf9596f39f80 (127.2.49.136) to b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132). New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:35.664358 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:35.665530 16781 tablet_service.cc:1514] Processing DeleteTablet for tablet dd11460051ab40febc2fe6aff58b5028 with delete_type TABLET_DATA_TOMBSTONED (TS 036223b0890d4634acfadf9596f39f80 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37770
I20240503 08:15:35.683944 27601 tablet_replica.cc:330] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: stopping tablet replica
I20240503 08:15:35.684916 27601 raft_consensus.cc:2229] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:35.685281 27601 raft_consensus.cc:2258] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:35.692082 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:35.693990 27601 ts_tablet_manager.cc:1875] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:35.701130 27601 ts_tablet_manager.cc:1888] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:15:35.701606 27601 log.cc:1196] T dd11460051ab40febc2fe6aff58b5028 P 036223b0890d4634acfadf9596f39f80: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd11460051ab40febc2fe6aff58b5028
I20240503 08:15:35.702625 14335 catalog_manager.cc:4830] TS 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): tablet dd11460051ab40febc2fe6aff58b5028 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:35.796727 14619 consensus_queue.cc:237] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:15:35.798802 16403 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:35.799274 14878 raft_consensus.cc:1261] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:35.799377 27206 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:35.799930 27206 consensus_queue.cc:1035] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:35.810629 27568 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:35.815207 14880 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:35.824108 14340 catalog_manager.cc:5484] T f7a5221747794a13aa4ed763006ba291 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:35.837183 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a5221747794a13aa4ed763006ba291 with delete_type TABLET_DATA_TOMBSTONED (TS 6835ad958e094ce4927135b942a2e8a8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54464
I20240503 08:15:35.841045 16403 raft_consensus.cc:2932] T f7a5221747794a13aa4ed763006ba291 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:15:35.891291 27631 tablet_replica.cc:330] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: stopping tablet replica
I20240503 08:15:35.893365 27631 raft_consensus.cc:2229] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:35.893947 27631 raft_consensus.cc:2258] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:35.898660 27631 ts_tablet_manager.cc:1875] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:35.919692 27631 ts_tablet_manager.cc:1888] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:35.922251 27631 log.cc:1196] T f7a5221747794a13aa4ed763006ba291 P 6835ad958e094ce4927135b942a2e8a8: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/wals/f7a5221747794a13aa4ed763006ba291
I20240503 08:15:35.925727 14335 catalog_manager.cc:4830] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): tablet f7a5221747794a13aa4ed763006ba291 (table rebalance_test_table_8 [id=7184b9ddb57b427cb18c0e1bbf24e3a1]) successfully deleted
I20240503 08:15:35.983947 14880 consensus_queue.cc:237] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:15:35.989498 14490 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:35.991344 16403 raft_consensus.cc:1261] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:35.992581 27589 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:35.993257 27589 consensus_queue.cc:1035] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:36.029382 27585 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:36.033383 16403 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:36.038776 14748 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:15:36.039567 27399 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:36.042969 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:36.046571 14344 catalog_manager.cc:5484] T 5d16c2981213464cbefa44b6ca509182 P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:36.051833 16516 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d16c2981213464cbefa44b6ca509182 with delete_type TABLET_DATA_TOMBSTONED (TS 95860e22c9474e2d894f04deba8442cc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43672
I20240503 08:15:36.061297 14489 raft_consensus.cc:2932] T 5d16c2981213464cbefa44b6ca509182 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:36.082880 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:36.085996 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:36.088140 27683 tablet_replica.cc:330] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: stopping tablet replica
I20240503 08:15:36.089864 27683 raft_consensus.cc:2229] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:36.090281 27683 raft_consensus.cc:2258] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:36.094329 16801 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:15:36.097869 27683 ts_tablet_manager.cc:1875] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:36.098222 27399 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:36.109196 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:36.120148 14619 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:15:36.120864 27585 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:36.138386 27683 ts_tablet_manager.cc:1888] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:15:36.138548 27683 log.cc:1196] T 5d16c2981213464cbefa44b6ca509182 P 95860e22c9474e2d894f04deba8442cc: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-5/wal/wals/5d16c2981213464cbefa44b6ca509182
I20240503 08:15:36.141577 14336 catalog_manager.cc:4830] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): tablet 5d16c2981213464cbefa44b6ca509182 (table rebalance_test_table_6 [id=5224e66c2cf144369fb29255cdd25306]) successfully deleted
I20240503 08:15:36.150102 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:36.186878 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:36.199561 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:36.252522 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:36.587973 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:36.609009 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:36.626806 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:36.662979 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:36.693878 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:36.748667 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:36.750516 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:36.752137 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:36.788704 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:36.791070 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:36.814847 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:36.834233 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:36.865419 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:36.916800 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:36.979717 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:37.001055 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:37.163340 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:37.193609 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:37.224293 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:37.228148 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:37.246779 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:37.296669 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:37.297034 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:37.335178 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:37.421072 14748 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
W20240503 08:15:37.431989 14686 consensus_peers.cc:485] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c130eb7ea8a74f4b8d9b79e23af70067. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:37.433508 14619 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:37.434248 27826 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:37.443114 16801 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:37.443996 27826 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:37.454564 27826 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.458506 14621 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.461577 16404 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:15:37.463882 14342 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.467135 14748 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:37.468835 22114 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:37.470718 14621 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 0f8702fad8ea4d22840dab11bb27f7f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:15:37.471311 22114 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:37.479056 16801 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.525290 27885 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:37.533310 14749 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:37.541766 14340 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 2 leader_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:37.547329 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet f65ab6dba1124573962e2da58aa1859c with delete_type TABLET_DATA_TOMBSTONED (TS 6835ad958e094ce4927135b942a2e8a8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54464
I20240503 08:15:37.550344 14620 raft_consensus.cc:2932] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:15:37.577163 27900 tablet_replica.cc:330] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: stopping tablet replica
I20240503 08:15:37.577513 27900 raft_consensus.cc:2229] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:37.577845 27900 raft_consensus.cc:2258] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:37.579154 27900 ts_tablet_manager.cc:1875] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:37.593055 27900 ts_tablet_manager.cc:1888] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:15:37.593300 27900 log.cc:1196] T f65ab6dba1124573962e2da58aa1859c P 6835ad958e094ce4927135b942a2e8a8: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-6/wal/wals/f65ab6dba1124573962e2da58aa1859c
I20240503 08:15:37.594125 14335 catalog_manager.cc:4830] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): tablet f65ab6dba1124573962e2da58aa1859c (table rebalance_test_table_0 [id=afbaaee2198d4c62a28b88392a8b6e06]) successfully deleted
I20240503 08:15:37.669135 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:37.699304 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:15:37.726365 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:37.738423 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:37.760360 16649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:37.800067 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:37.815973 14490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:37.839102 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:37.841970 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:37.849009 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:37.862653 14815 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:37.864252 14489 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:37.865028 27399 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:37.882917 16801 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:37.883659 27399 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:37.883828 16535 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:37.884402 27582 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:37.885381 14620 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:15:37.886646 14425 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:37.886760 27806 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:37.907380 27582 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.909802 16536 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 95860e22c9474e2d894f04deba8442cc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.917380 14620 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.922425 14349 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:37.961133 27399 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:37.962877 27955 ts_tablet_manager.cc:902] T c130eb7ea8a74f4b8d9b79e23af70067 P 6835ad958e094ce4927135b942a2e8a8: Initiating tablet copy from peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
F20240503 08:15:37.963201 27955 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.queue_time_us, tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.queue_time_us) (tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_cpu_time_us, tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_cpu_time_us) (tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_wall_time_us, tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_wall_time_us) (tablet-download-pool-1c687520c0a349fd8438dacbd5011001.queue_time_us, tablet-download-pool-1c687520c0a349fd8438dacbd5011001.queue_time_us) (tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_cpu_time_us, tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_cpu_time_us) (tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_wall_time_us, tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_wall_time_us) (tablet-download-pool-30b13cdf4b234f17872f5d1588915968.queue_time_us, tablet-download-pool-30b13cdf4b234f17872f5d1588915968.queue_time_us) (tablet-download-pool-30b13cdf4b234f17872f5d1588915968.run_cpu_time_us, tablet-download-pool-30b13cdf4b234f17872f5d1588915968.run_cpu_time_us) (tablet-download-pool-30b13cdf4b234f17872f5d1588915968.run_wall_time_us, tablet-download-pool-30b13cdf4b234f17872f5d1588915968.run_wall_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.queue_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.queue_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_cpu_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_cpu_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_wall_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_wall_time_us) (tablet-download-pool-55067e08fb694e90b6366d41cc962095.queue_time_us, tablet-download-pool-55067e08fb694e90b6366d41cc962095.queue_time_us) (tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_cpu_time_us, tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_cpu_time_us) (tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_wall_time_us, tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_wall_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.queue_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.queue_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.run_cpu_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.run_cpu_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.run_wall_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.run_wall_time_us) (tablet-download-pool-76a216295a8a4feaa2d3326035879073.queue_time_us, tablet-download-pool-76a216295a8a4feaa2d3326035879073.queue_time_us) (tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_cpu_time_us, tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_cpu_time_us) (tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_wall_time_us, tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_wall_time_us) (tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.queue_time_us, tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.queue_time_us) (tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_cpu_time_us, tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_cpu_time_us) (tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_wall_time_us, tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_wall_time_us) (tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.queue_time_us, tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.queue_time_us) (tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_cpu_time_us, tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_cpu_time_us) (tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_wall_time_us, tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_wall_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.queue_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.queue_time_us) (tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.queue_time_us, tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.queue_time_us) (tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_cpu_time_us, tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_cpu_time_us) (tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_wall_time_us, tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_wall_time_us) (tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.queue_time_us, tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.queue_time_us) (tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.run_cpu_time_us, tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.run_cpu_time_us) (tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.run_wall_time_us, tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.run_wall_time_us) (tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.queue_time_us, tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.queue_time_us) (tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_cpu_time_us, tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_cpu_time_us) (tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_wall_time_us, tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_wall_time_us) (tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.queue_time_us, tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.queue_time_us) (tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_cpu_time_us, tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_cpu_time_us) (tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_wall_time_us, tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_wall_time_us) (tablet-download-pool-f7a5221747794a13aa4ed763006ba291.queue_time_us, tablet-download-pool-f7a5221747794a13aa4ed763006ba291.queue_time_us) (tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_cpu_time_us, tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_cpu_time_us) (tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_wall_time_us, tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 08:15:37.964831 14490 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
    @     0x7f8ceed15dcd  google::LogMessage::Fail() at ??:0
I20240503 08:15:37.974460 14340 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:15:37.980942 16801 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
    @     0x7f8ceed19b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f8ceed157cc  google::LogMessage::Flush() at ??:0
    @     0x7f8ceed16f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f8cf1a56742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f8cf1a22f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f8cf1a1ccd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f8d00e708de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f8d00e88daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f8d00fb3480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f8d00feba8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f8d00fce981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f8d00fce70e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f8cf1a26b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f8cf1a2a19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f8cf1a2a019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f8cf1a0501e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f8cf4d8a6db  start_thread at ??:0
    @     0x7f8cedbc871f  clone at ??:0
I20240503 08:15:38.275585 14880 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:15:38.288443 14878 consensus_queue.cc:237] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:38.291074 14620 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:15:38.291344 14749 raft_consensus.cc:1261] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:15:38.291790 27812 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:38.295068 27812 consensus_queue.cc:1035] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:15:38.307528 14817 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc9575f64eb741948e51ccf06f48228a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:38.308351 14748 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:15:38.308861 14619 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:15:38.309772 27812 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:38.312480 27812 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:38.365176 28051 ts_tablet_manager.cc:902] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:15:38.371058 27812 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:38.379760 14619 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:38.383148 14340 catalog_manager.cc:5484] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 11 to 13, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:38.387887 16781 tablet_service.cc:1514] Processing DeleteTablet for tablet 169bd9ea6772420abf757dd55cf44d31 with delete_type TABLET_DATA_TOMBSTONED (TS 036223b0890d4634acfadf9596f39f80 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:37770
I20240503 08:15:38.389366 14749 raft_consensus.cc:2932] T 169bd9ea6772420abf757dd55cf44d31 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:15:38.397939 27399 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:38.404139 28051 tablet_copy_client.cc:308] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:15:38.403690 14620 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:38.405077 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 72a949e76ab5477680a4f9912e9672e9 from peer 0f8702fad8ea4d22840dab11bb27f7f2 ({username='slave'} at 127.2.49.133:60169)
I20240503 08:15:38.405218 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 72a949e76ab5477680a4f9912e9672e9 from peer 0f8702fad8ea4d22840dab11bb27f7f2 at {username='slave'} at 127.2.49.133:60169: session id = 0f8702fad8ea4d22840dab11bb27f7f2-72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:38.405517 14748 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:38.406458 14340 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 11 to 12, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) added. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } }
I20240503 08:15:38.407979 14901 tablet_copy_source_session.cc:205] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:38.409408 28051 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:38.426987 28055 tablet_replica.cc:330] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: stopping tablet replica
I20240503 08:15:38.427263 28055 raft_consensus.cc:2229] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:38.427459 28055 raft_consensus.cc:2258] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:38.428273 28055 ts_tablet_manager.cc:1875] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:38.437930 28051 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.13
I20240503 08:15:38.439078 28051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a949e76ab5477680a4f9912e9672e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:38.443984 28051 tablet_copy_client.cc:783] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:38.444289 28051 tablet_copy_client.cc:647] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:38.446388 28051 tablet_copy_client.cc:523] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:38.456029 28051 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap starting.
I20240503 08:15:38.468919 28055 ts_tablet_manager.cc:1888] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.12
I20240503 08:15:38.469056 28055 log.cc:1196] T 169bd9ea6772420abf757dd55cf44d31 P 036223b0890d4634acfadf9596f39f80: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-7/wal/wals/169bd9ea6772420abf757dd55cf44d31
I20240503 08:15:38.469934 14335 catalog_manager.cc:4830] TS 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): tablet 169bd9ea6772420abf757dd55cf44d31 (table rebalance_test_table_1 [id=6815d5df4efd45d59a2db67226231af0]) successfully deleted
I20240503 08:15:38.728143 28113 ts_tablet_manager.cc:902] T dc9575f64eb741948e51ccf06f48228a P 95860e22c9474e2d894f04deba8442cc: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
F20240503 08:15:38.729614 28113 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1c687520c0a349fd8438dacbd5011001.queue_time_us, tablet-download-pool-1c687520c0a349fd8438dacbd5011001.queue_time_us) (tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_cpu_time_us, tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_cpu_time_us) (tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_wall_time_us, tablet-download-pool-1c687520c0a349fd8438dacbd5011001.run_wall_time_us) (tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.queue_time_us, tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.queue_time_us) (tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_cpu_time_us, tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_cpu_time_us) (tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_wall_time_us, tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_wall_time_us) (tablet-download-pool-55067e08fb694e90b6366d41cc962095.queue_time_us, tablet-download-pool-55067e08fb694e90b6366d41cc962095.queue_time_us) (tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_cpu_time_us, tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_cpu_time_us) (tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_wall_time_us, tablet-download-pool-55067e08fb694e90b6366d41cc962095.run_wall_time_us) (tablet-download-pool-5c595bb19d6946b0b1f853064892e946.queue_time_us, tablet-download-pool-5c595bb19d6946b0b1f853064892e946.queue_time_us) (tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_cpu_time_us, tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_cpu_time_us) (tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_wall_time_us, tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_wall_time_us) (tablet-download-pool-5d16c2981213464cbefa44b6ca509182.queue_time_us, tablet-download-pool-5d16c2981213464cbefa44b6ca509182.queue_time_us) (tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_cpu_time_us, tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_cpu_time_us) (tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_wall_time_us, tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_wall_time_us) (tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.queue_time_us, tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.queue_time_us) (tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_cpu_time_us, tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_cpu_time_us) (tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_wall_time_us, tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_wall_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.queue_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.queue_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_cpu_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_cpu_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_wall_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_wall_time_us) (tablet-download-pool-76a216295a8a4feaa2d3326035879073.queue_time_us, tablet-download-pool-76a216295a8a4feaa2d3326035879073.queue_time_us) (tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_cpu_time_us, tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_cpu_time_us) (tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_wall_time_us, tablet-download-pool-76a216295a8a4feaa2d3326035879073.run_wall_time_us) (tablet-download-pool-7cc0fe7d81e149d5b82fabfdc11d66b4.queue_time_us, tablet-download-pool-7cc0fe7d81e149d5b82fabfdc11d66b4.queue_time_us) (tablet-download-pool-7cc0fe7d81e149d5b82fabfdc11d66b4.run_cpu_time_us, tablet-download-pool-7cc0fe7d81e149d5b82fabfdc11d66b4.run_cpu_time_us) (tablet-download-pool-7cc0fe7d81e149d5b82fabfdc11d66b4.run_wall_time_us, tablet-download-pool-7cc0fe7d81e149d5b82fabfdc11d66b4.run_wall_time_us) (tablet-download-pool-897a701b880641b08816f2d1a254ef03.queue_time_us, tablet-download-pool-897a701b880641b08816f2d1a254ef03.queue_time_us) (tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_cpu_time_us, tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_cpu_time_us) (tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_wall_time_us, tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_wall_time_us) (tablet-download-pool-b5ceefc466f642d08ea96d549a534776.queue_time_us, tablet-download-pool-b5ceefc466f642d08ea96d549a534776.queue_time_us) (tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_cpu_time_us, tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_cpu_time_us) (tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_wall_time_us, tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_wall_time_us) (tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.queue_time_us, tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.queue_time_us) (tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_cpu_time_us, tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_cpu_time_us) (tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_wall_time_us, tablet-download-pool-da1f165b4a5647a48ad7ff951fb3fbf6.run_wall_time_us) (tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.queue_time_us, tablet-download-pool-dc9575f64eb741948e51ccf06f48228a.queue_time_us) (tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.queue_time_us, tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.queue_time_us) (tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_cpu_time_us, tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_cpu_time_us) (tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_wall_time_us, tablet-download-pool-dfa711e8c8564bc78846b16d68470cea.run_wall_time_us) (tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.queue_time_us, tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.queue_time_us) (tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_cpu_time_us, tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_cpu_time_us) (tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_wall_time_us, tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f664d19cdcd  google::LogMessage::Fail() at ??:0
I20240503 08:15:38.739035 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
    @     0x7f664d1a0b93  google::LogMessage::SendToLog() at ??:0
I20240503 08:15:38.767481 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7f664d19c7cc  google::LogMessage::Flush() at ??:0
I20240503 08:15:38.784282 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
    @     0x7f664d19df59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:15:38.803226 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:38.804394 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
    @     0x7f664fedd742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f664fea9f8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240503 08:15:38.831445 16516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7f664fea3cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f665f2f78de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f665f30fdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:15:38.882685 28051 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:38.883167 28051 tablet_bootstrap.cc:492] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Bootstrap complete.
I20240503 08:15:38.883564 28051 ts_tablet_manager.cc:1367] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent bootstrapping tablet: real 0.428s	user 0.078s	sys 0.000s
I20240503 08:15:38.884431 28051 raft_consensus.cc:348] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:38.884670 28051 raft_consensus.cc:729] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8702fad8ea4d22840dab11bb27f7f2, State: Initialized, Role: FOLLOWER
I20240503 08:15:38.885105 14599 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
    @     0x7f665f43a480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:15:38.892943 28051 consensus_queue.cc:260] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:15:38.893548 28051 ts_tablet_manager.cc:1398] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20240503 08:15:38.897259 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-72a949e76ab5477680a4f9912e9672e9 received from {username='slave'} at 127.2.49.133:60169
I20240503 08:15:38.897387 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 0f8702fad8ea4d22840dab11bb27f7f2-72a949e76ab5477680a4f9912e9672e9 on tablet 72a949e76ab5477680a4f9912e9672e9 with peer 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:15:38.909018 14489 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } }
I20240503 08:15:38.915037 16404 raft_consensus.cc:1203] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.14->[2.15-2.15]   Dedup: 2.15->[]
    @     0x7f665f472a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 08:15:38.925583 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:38.933023 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:38.939028 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:38.954872 14620 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:15:38.955440 27582 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:38.960184 16404 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:15:38.960752 27582 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
    @     0x7f665f455981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f665f45570e  std::_Function_handler<>::_M_invoke() at ??:0
I20240503 08:15:39.004527 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f664feadb75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f664feb119c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f664feb1019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f664fe8c01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f66532116db  start_thread at ??:0
    @     0x7f664c04f71f  clone at ??:0
I20240503 08:15:39.083292 27582 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:39.086895 14619 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:39.100507 14349 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:39.115056 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:39.114730 16404 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } }
I20240503 08:15:39.153846 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:39.217645 27582 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6835ad958e094ce4927135b942a2e8a8)
I20240503 08:15:39.217831 27582 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:39.218600 27582 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299), 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
I20240503 08:15:39.220288 14620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
W20240503 08:15:39.327778 14556 connection.cc:535] client connection to 127.2.49.135:46299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.328056 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.328150 14686 connection.cc:535] client connection to 127.2.49.135:46299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.328341 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.335203 14425 connection.cc:535] client connection to 127.2.49.135:46299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.335500 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.361524 28179 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:39.329637 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.135:46299 (local address 127.2.49.130:40481)
0503 08:15:39.334138 (+  4501us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:39.334179 (+    41us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:39.334380 (+   201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:39.360295 (+ 25915us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4362,"thread_start_us":4276,"threads_started":1}
W20240503 08:15:39.361824 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.361997 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.376852 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.377000 14425 leader_election.cc:334] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:39.377097 14425 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 0d77c4b1f12049dea71b97e86be2b288, 6835ad958e094ce4927135b942a2e8a8
W20240503 08:15:39.377499 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:39.377718 27582 raft_consensus.cc:2726] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240503 08:15:39.380160 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.381129 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.382913 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.383046 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.390450 28183 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:39.338903 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.135:46299 (local address 127.2.49.131:36789)
0503 08:15:39.388109 (+ 49206us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:39.388280 (+   171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48986,"thread_start_us":48956,"threads_started":1}
W20240503 08:15:39.390621 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection reset by peer (error 104)
W20240503 08:15:39.390708 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:39.420523 27568 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6835ad958e094ce4927135b942a2e8a8)
I20240503 08:15:39.420751 27568 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:39.421828 27568 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:15:39.423650 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" is_pre_election: true
I20240503 08:15:39.424096 14489 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 3.
W20240503 08:15:39.428579 14556 leader_election.cc:334] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:39.430617 14557 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 6835ad958e094ce4927135b942a2e8a8
I20240503 08:15:39.435571 27568 raft_consensus.cc:2781] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 08:15:39.435712 27568 raft_consensus.cc:482] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6835ad958e094ce4927135b942a2e8a8)
I20240503 08:15:39.435794 27568 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:39.444872 27568 raft_consensus.cc:504] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:15:39.449836 27568 leader_election.cc:288] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 4 election: Requested vote from peers 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299), 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
W20240503 08:15:39.452069 14556 leader_election.cc:334] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:39.488237 14557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.49.129:37829, user_credentials={real_user=slave}} blocked reactor thread for 59625.6us
I20240503 08:15:39.489065 14489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86f4f3a01048e69f553c6557d79b71" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f"
I20240503 08:15:39.489279 14489 raft_consensus.cc:3037] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:15:39.492322 14489 raft_consensus.cc:2454] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 4.
I20240503 08:15:39.495714 14557 leader_election.cc:302] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, 129a2ac87d7f416db6d47d79bb1c2b5f; no voters: 6835ad958e094ce4927135b942a2e8a8
I20240503 08:15:39.496425 28202 raft_consensus.cc:2781] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:15:39.497109 28202 raft_consensus.cc:686] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 4 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:15:39.497468 28202 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
W20240503 08:15:39.514554 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:39.520397 14342 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 3 to 4, leader changed from 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 4 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6835ad958e094ce4927135b942a2e8a8" member_type: VOTER last_known_addr { host: "127.2.49.135" port: 46299 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
W20240503 08:15:39.567260 14687 connection.cc:535] client connection to 127.2.49.134:36937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.568125 14426 connection.cc:535] client connection to 127.2.49.134:36937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.573207 14817 connection.cc:535] client connection to 127.2.49.134:36937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.573318 14557 connection.cc:535] client connection to 127.2.49.134:36937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.573463 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:15:39.574501 28183 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:39.568366 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.134:36937 (local address 127.2.49.131:52759)
0503 08:15:39.568719 (+   353us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:39.568743 (+    24us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:39.568920 (+   177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:39.574329 (+  5409us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":184}
W20240503 08:15:39.574736 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.574838 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.574992 28185 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:39.569332 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.134:36937 (local address 127.2.49.129:52563)
0503 08:15:39.569738 (+   406us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:39.569759 (+    21us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:39.569899 (+   140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:39.574085 (+  4186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":264}
W20240503 08:15:39.574973 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575081 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575196 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575297 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575364 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575423 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575479 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.575556 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:15:39.608000 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:39.608168 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:39.756613 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:39.764401 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:39.790747 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:39.850960 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:39.900003 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:39.914310 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:39.916918 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:39.929313 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:39.962708 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:39.963371 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:39.983376 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:40.005288 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:40.020270 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:40.027153 14489 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 4 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240503 08:15:40.027838 28202 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 08:15:40.072715 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:40.074203 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:40.088224 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:40.432803 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:40.450579 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:40.464478 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:40.467610 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:40.469558 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:40.485782 16853 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 08:14:29.420596 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:29.421092 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:29.456332 16859 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:33.570317 16859 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:14:35.286955 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.290565 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.290696 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.290803 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:35.290923 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.291337 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.291469 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.291625 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.291728 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:35.291957 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.292153 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.292282 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.292385 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.292511 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.292614 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:35.837705 16859 rebalancer_tool.cc:1074] tablet dc9575f64eb741948e51ccf06f48228a: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:36.169814 16859 rebalancer_tool.cc:1074] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:36.933550 16859 rebalancer_tool.cc:1074] tablet a87c5af03f4d47d38bbfbd225c7195d1: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:37.711005 16859 rebalancer_tool.cc:1074] tablet 3fea213e5a12452f8b894b9cdb94f9ae: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:38.190341 16859 rebalancer_tool.cc:1074] tablet 4e86f4f3a01048e69f553c6557d79b71: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:38.654086 16859 rebalancer_tool.cc:1074] tablet b5ceefc466f642d08ea96d549a534776: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:38.732218 16859 rebalancer_tool.cc:1084] tablet 72a949e76ab5477680a4f9912e9672e9: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move ignored: Illegal state: tablet 72a949e76ab5477680a4f9912e9672e9: replica 0f8702fad8ea4d22840dab11bb27f7f2 is already present
I20240503 08:14:38.732453 16859 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:40.558737 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:40.558959 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.559398 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.559542 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.559739 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:40.559983 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.560115 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:40.560393 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.560504 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:40.560599 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.560743 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:40.560840 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:40.561005 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:40.640172 16859 rebalancer_tool.cc:1074] tablet 5c595bb19d6946b0b1f853064892e946: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:41.116276 16859 rebalancer_tool.cc:1074] tablet 12df78f752c34fe3885b235c4a19e325: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:41.480688 16859 rebalancer_tool.cc:1074] tablet 1d6c920958214148b3c3d5ac995e99e2: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:41.675189 16859 rebalancer_tool.cc:1074] tablet 897a701b880641b08816f2d1a254ef03: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:14:42.459698 16859 rebalancer_tool.cc:1074] tablet 7aa1031b9e30478597da5ad3708bdff5: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:42.815368 16859 rebalancer_tool.cc:1074] tablet f19b7ef418974b06bdcc7125e8cf732d: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:42.964259 16859 rebalancer_tool.cc:1074] tablet 72a949e76ab5477680a4f9912e9672e9: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:43.350399 16859 rebalancer_tool.cc:1074] tablet 5d16c2981213464cbefa44b6ca509182: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:43.897977 16859 rebalancer_tool.cc:1074] tablet 6b677894b51f4dc58f157d8f4d632e9a: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:44.356525 16859 rebalancer_tool.cc:1074] tablet 562e1ec5c34945298cf457758328057f: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:44.443903 16859 rebalancer_tool.cc:1074] tablet f65ab6dba1124573962e2da58aa1859c: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:44.580230 16859 rebalancer_tool.cc:1084] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move ignored: Illegal state: tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: replica 95860e22c9474e2d894f04deba8442cc is already present
I20240503 08:14:44.580674 16859 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:46.622823 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:46.623028 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:46.623134 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:46.623315 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:46.623420 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:46.623625 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:46.623759 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:46.623838 16859 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:46.759811 16859 rebalancer_tool.cc:1074] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:53.898947 16859 rebalancer_tool.cc:1136] tablet dc9575f64eb741948e51ccf06f48228a: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:02.158313 16859 rebalancer_tool.cc:1136] tablet 5c595bb19d6946b0b1f853064892e946: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:08.538496 16859 rebalancer_tool.cc:1136] tablet a87c5af03f4d47d38bbfbd225c7195d1: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:08.654461 16859 rebalancer_tool.cc:1136] tablet 897a701b880641b08816f2d1a254ef03: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:08.676159 16859 rebalancer_tool.cc:1136] tablet 7aa1031b9e30478597da5ad3708bdff5: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:10.060624 16859 rebalancer_tool.cc:1123] tablet f19b7ef418974b06bdcc7125e8cf732d: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '95860e22c9474e2d894f04deba8442cc' move is abandoned: Not found: Server with UUID b35c4c5d528c4a8f9e754ff9481c3938 not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:15:10.141407 16859 rebalancer_tool.cc:1136] tablet 72a949e76ab5477680a4f9912e9672e9: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:14.540831 16859 rebalancer_tool.cc:1074] tablet 169bd9ea6772420abf757dd55cf44d31: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:14.735324 16859 rebalancer_tool.cc:1074] tablet c130eb7ea8a74f4b8d9b79e23af70067: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:14.948798 16859 rebalancer_tool.cc:1074] tablet 30b13cdf4b234f17872f5d1588915968: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:15.143703 16859 rebalancer_tool.cc:1136] tablet c130eb7ea8a74f4b8d9b79e23af70067: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:15.249022 16859 rebalancer_tool.cc:1136] tablet 169bd9ea6772420abf757dd55cf44d31: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:15.338086 16859 rebalancer_tool.cc:1136] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:15.418185 16859 rebalancer_tool.cc:1136] tablet 3fea213e5a12452f8b894b9cdb94f9ae: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:15.734632 16859 rebalancer_tool.cc:1136] tablet b5ceefc466f642d08ea96d549a534776: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:15.781994 16859 rebalancer_tool.cc:1136] tablet f65ab6dba1124573962e2da58aa1859c: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:15.951850 16859 rebalancer_tool.cc:1136] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:16.050357 16859 rebalancer_tool.cc:1136] tablet 4e86f4f3a01048e69f553c6557d79b71: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:16.110760 16859 rebalancer_tool.cc:1136] tablet 12df78f752c34fe3885b235c4a19e325: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:16.244388 16859 rebalancer_tool.cc:1136] tablet 1d6c920958214148b3c3d5ac995e99e2: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:16.341360 16859 rebalancer_tool.cc:1136] tablet 6b677894b51f4dc58f157d8f4d632e9a: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:18.249373 16859 rebalancer_tool.cc:1136] tablet 5d16c2981213464cbefa44b6ca509182: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:18.377562 16859 rebalancer_tool.cc:1136] tablet 562e1ec5c34945298cf457758328057f: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:18.485175 16859 rebalancer_tool.cc:1074] tablet dfa711e8c8564bc78846b16d68470cea: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:18.565896 16859 rebalancer_tool.cc:1074] tablet f7a5221747794a13aa4ed763006ba291: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:18.719121 16859 rebalancer_tool.cc:1074] tablet 0780fb979eee4bae98a014d9721c9441: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:18.970978 16859 rebalancer_tool.cc:1074] tablet 1c687520c0a349fd8438dacbd5011001: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:19.128671 16859 rebalancer_tool.cc:1136] tablet 1c687520c0a349fd8438dacbd5011001: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:21.038870 16859 rebalancer_tool.cc:1136] tablet f7a5221747794a13aa4ed763006ba291: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:21.179431 16859 rebalancer_tool.cc:1136] tablet dfa711e8c8564bc78846b16d68470cea: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:21.391130 16859 rebalancer_tool.cc:1136] tablet 30b13cdf4b234f17872f5d1588915968: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:21.483060 16859 rebalancer_tool.cc:1074] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:21.579542 16859 rebalancer_tool.cc:1136] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:23.715103 16859 rebalancer_tool.cc:1136] tablet 0780fb979eee4bae98a014d9721c9441: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:26.174394 16859 rebalancer_tool.cc:1074] tablet f7a5221747794a13aa4ed763006ba291: '6835ad958e094ce4927135b942a2e8a8' -> 'b35c4c5d528c4a8f9e754ff9481c3938' move scheduled
I20240503 08:15:26.242687 16859 rebalancer_tool.cc:1074] tablet 897a701b880641b08816f2d1a254ef03: '95860e22c9474e2d894f04deba8442cc' -> '' move scheduled
I20240503 08:15:26.336611 16859 rebalancer_tool.cc:1084] tablet 4e86f4f3a01048e69f553c6557d79b71: '0f8702fad8ea4d22840dab11bb27f7f2' -> '129a2ac87d7f416db6d47d79bb1c2b5f' move ignored: Illegal state: tablet 4e86f4f3a01048e69f553c6557d79b71: replica 129a2ac87d7f416db6d47d79bb1c2b5f is already present
I20240503 08:15:26.336774 16859 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:15:27.514636 26285 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.2.49.190:42751, user_credentials={real_user=slave}} blocked reactor thread for 58483.5us
I20240503 08:15:28.724586 16859 rebalancer_tool.cc:1074] tablet dc9575f64eb741948e51ccf06f48228a: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move scheduled
I20240503 08:15:29.456651 16859 rebalancer_tool.cc:1074] tablet 4e86f4f3a01048e69f553c6557d79b71: '0f8702fad8ea4d22840dab11bb27f7f2' -> '0d77c4b1f12049dea71b97e86be2b288' move scheduled
I20240503 08:15:34.258746 16859 rebalancer_tool.cc:1136] tablet 897a701b880641b08816f2d1a254ef03: '95860e22c9474e2d894f04deba8442cc' -> '' move completed: OK
I20240503 08:15:35.898813 16859 rebalancer_tool.cc:1136] tablet 4e86f4f3a01048e69f553c6557d79b71: '0f8702fad8ea4d22840dab11bb27f7f2' -> '0d77c4b1f12049dea71b97e86be2b288' move completed: OK
I20240503 08:15:36.176944 16859 rebalancer_tool.cc:1136] tablet dc9575f64eb741948e51ccf06f48228a: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move completed: OK
I20240503 08:15:36.330651 16859 rebalancer_tool.cc:1136] tablet f7a5221747794a13aa4ed763006ba291: '6835ad958e094ce4927135b942a2e8a8' -> 'b35c4c5d528c4a8f9e754ff9481c3938' move completed: OK
I20240503 08:15:37.925253 16859 rebalancer_tool.cc:1074] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:15:37.994129 16859 rebalancer_tool.cc:1074] tablet f65ab6dba1124573962e2da58aa1859c: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:38.055570 16859 rebalancer_tool.cc:1084] tablet 72a949e76ab5477680a4f9912e9672e9: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move ignored: Illegal state: tablet 72a949e76ab5477680a4f9912e9672e9: replica 0f8702fad8ea4d22840dab11bb27f7f2 is already present
I20240503 08:15:38.055752 16859 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:15:39.344461 28129 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:38.996591 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.135:46299 (local address 127.0.0.1:53480)
0503 08:15:38.996785 (+   194us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:38.996807 (+    22us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:38.996912 (+   105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:39.343006 (+346094us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20240503 08:15:39.573689 28131 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:39.062695 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.134:36937 (local address 127.0.0.1:52544)
0503 08:15:39.062855 (+   160us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:39.062876 (+    21us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:39.063037 (+   161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:39.573463 (+510426us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
Illegal state: tablet server 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): unacceptable health status UNAVAILABLE

I20240503 08:15:40.511274 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:40.563975 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:40.582456 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:40.646009 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:40.784039 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:40.806304 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:40.806921 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:40.843688 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:40.858253 14880 consensus_queue.cc:237] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
W20240503 08:15:40.869580 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:40.873021 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:40.893697 16800 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240503 08:15:40.894531 28056 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 08:15:40.911950 28056 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:40.914916 16800 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:40.924865 14342 catalog_manager.cc:5484] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 15 to 16, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New cstate: current_term: 2 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:40.939286 14489 raft_consensus.cc:1261] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240503 08:15:40.972007 28056 consensus_queue.cc:1035] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 08:15:41.018577 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:41.020606 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:41.025188 14489 raft_consensus.cc:2932] T 72a949e76ab5477680a4f9912e9672e9 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:15:41.035429 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a949e76ab5477680a4f9912e9672e9 with delete_type TABLET_DATA_TOMBSTONED (TS 0f8702fad8ea4d22840dab11bb27f7f2 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:43360
I20240503 08:15:41.049180 28384 tablet_replica.cc:330] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: stopping tablet replica
I20240503 08:15:41.049451 28384 raft_consensus.cc:2229] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:41.049636 28384 raft_consensus.cc:2258] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:41.050586 28384 ts_tablet_manager.cc:1875] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:41.057026 16848 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 08:14:28.984349 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:28.984963 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:29.003319 16856 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:32.515918 16856 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:14:34.432938 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:34.433229 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:34.433529 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:34.434140 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:34.434301 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:34.434532 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:34.434649 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:34.435026 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:34.435262 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:34.435461 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:34.435570 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:34.435714 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:34.435817 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:34.435942 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:34.858747 17629 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.2.49.190:42751, user_credentials={real_user=slave}} blocked reactor thread for 68130.5us
I20240503 08:14:35.212596 16856 rebalancer_tool.cc:1074] tablet 897a701b880641b08816f2d1a254ef03: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:35.778182 16856 rebalancer_tool.cc:1074] tablet 30b13cdf4b234f17872f5d1588915968: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:36.350651 16856 rebalancer_tool.cc:1074] tablet b5ceefc466f642d08ea96d549a534776: '0d77c4b1f12049dea71b97e86be2b288' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:36.793483 16856 rebalancer_tool.cc:1074] tablet 5c595bb19d6946b0b1f853064892e946: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:37.721015 16856 rebalancer_tool.cc:1074] tablet c130eb7ea8a74f4b8d9b79e23af70067: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:38.349236 16856 rebalancer_tool.cc:1074] tablet 76a216295a8a4feaa2d3326035879073: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:38.487310 16856 rebalancer_tool.cc:1084] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move ignored: Illegal state: tablet 55067e08fb694e90b6366d41cc962095: replica 95860e22c9474e2d894f04deba8442cc is already present
I20240503 08:14:38.487545 16856 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:40.680917 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.681226 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.694787 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.695001 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:40.695166 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.695282 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:40.695369 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.695477 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:40.695713 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:40.695863 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.695969 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:40.696100 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:40.696190 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:40.696307 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:40.864684 16856 rebalancer_tool.cc:1074] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:41.283195 16856 rebalancer_tool.cc:1074] tablet 1d6c920958214148b3c3d5ac995e99e2: '0d77c4b1f12049dea71b97e86be2b288' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:42.054075 16856 rebalancer_tool.cc:1074] tablet 5d16c2981213464cbefa44b6ca509182: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:42.340085 16856 rebalancer_tool.cc:1074] tablet dc9575f64eb741948e51ccf06f48228a: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:14:42.827896 16856 rebalancer_tool.cc:1084] tablet f19b7ef418974b06bdcc7125e8cf732d: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '036223b0890d4634acfadf9596f39f80' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } }
I20240503 08:14:42.828044 16856 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:44.304095 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:44.304478 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:44.304626 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:44.304986 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:44.305261 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:44.305491 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
I20240503 08:14:44.505534 16856 rebalancer_tool.cc:1074] tablet a87c5af03f4d47d38bbfbd225c7195d1: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:44.571851 16856 rebalancer_tool.cc:1074] tablet 6b677894b51f4dc58f157d8f4d632e9a: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:14:44.649578 16856 rebalancer_tool.cc:1074] tablet b19df3ae9f2b44a3818855f259962fff: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:14:58.136814 16856 rebalancer_tool.cc:1136] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:00.940755 16856 rebalancer_tool.cc:1136] tablet 30b13cdf4b234f17872f5d1588915968: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:01.049237 16856 rebalancer_tool.cc:1136] tablet b5ceefc466f642d08ea96d549a534776: '0d77c4b1f12049dea71b97e86be2b288' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:05.540115 16856 rebalancer_tool.cc:1136] tablet 5c595bb19d6946b0b1f853064892e946: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:07.709957 16856 rebalancer_tool.cc:1074] tablet 72a949e76ab5477680a4f9912e9672e9: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:07.788178 16856 rebalancer_tool.cc:1136] tablet 72a949e76ab5477680a4f9912e9672e9: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:07.830930 16856 rebalancer_tool.cc:1136] tablet b19df3ae9f2b44a3818855f259962fff: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:08.069077 16856 rebalancer_tool.cc:1136] tablet dc9575f64eb741948e51ccf06f48228a: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:08.218719 16856 rebalancer_tool.cc:1136] tablet 5d16c2981213464cbefa44b6ca509182: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:08.264894 16856 rebalancer_tool.cc:1136] tablet 6b677894b51f4dc58f157d8f4d632e9a: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:08.479952 16856 rebalancer_tool.cc:1136] tablet 1d6c920958214148b3c3d5ac995e99e2: '0d77c4b1f12049dea71b97e86be2b288' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:08.651942 16856 rebalancer_tool.cc:1136] tablet a87c5af03f4d47d38bbfbd225c7195d1: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:08.707544 16856 rebalancer_tool.cc:1136] tablet 897a701b880641b08816f2d1a254ef03: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:08.846212 16856 rebalancer_tool.cc:1136] tablet 76a216295a8a4feaa2d3326035879073: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:08.910472 16856 rebalancer_tool.cc:1136] tablet c130eb7ea8a74f4b8d9b79e23af70067: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:09.034651 16856 rebalancer_tool.cc:1074] tablet 7aa1031b9e30478597da5ad3708bdff5: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:09.128743 16856 rebalancer_tool.cc:1074] tablet dfa711e8c8564bc78846b16d68470cea: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:09.225587 16856 rebalancer_tool.cc:1074] tablet da1f165b4a5647a48ad7ff951fb3fbf6: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:09.426431 16856 rebalancer_tool.cc:1074] tablet 3fea213e5a12452f8b894b9cdb94f9ae: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:09.506929 16856 rebalancer_tool.cc:1074] tablet 169bd9ea6772420abf757dd55cf44d31: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
W20240503 08:15:12.212596 23707 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 117692us
I20240503 08:15:13.701730 16856 rebalancer_tool.cc:1136] tablet 169bd9ea6772420abf757dd55cf44d31: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:15.724769 16856 rebalancer_tool.cc:1136] tablet dfa711e8c8564bc78846b16d68470cea: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:15.840183 16856 rebalancer_tool.cc:1136] tablet 7aa1031b9e30478597da5ad3708bdff5: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:16.142844 16856 rebalancer_tool.cc:1074] tablet dd11460051ab40febc2fe6aff58b5028: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:16.278625 16856 rebalancer_tool.cc:1074] tablet 4e86f4f3a01048e69f553c6557d79b71: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:16.330555 16856 rebalancer_tool.cc:1074] tablet 562e1ec5c34945298cf457758328057f: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:16.399920 16856 rebalancer_tool.cc:1136] tablet 562e1ec5c34945298cf457758328057f: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:20.091089 16856 rebalancer_tool.cc:1136] tablet 3fea213e5a12452f8b894b9cdb94f9ae: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:20.387789 16856 rebalancer_tool.cc:1136] tablet dd11460051ab40febc2fe6aff58b5028: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:23.509673 16856 rebalancer_tool.cc:1074] tablet 1c687520c0a349fd8438dacbd5011001: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:23.550880 16856 rebalancer_tool.cc:1074] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:23.612195 16856 rebalancer_tool.cc:1074] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:23.698292 16856 rebalancer_tool.cc:1136] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:23.754987 16856 rebalancer_tool.cc:1136] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:23.830677 16856 rebalancer_tool.cc:1136] tablet 1c687520c0a349fd8438dacbd5011001: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:26.408104 16856 rebalancer_tool.cc:1136] tablet da1f165b4a5647a48ad7ff951fb3fbf6: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:26.553498 16856 rebalancer_tool.cc:1136] tablet 4e86f4f3a01048e69f553c6557d79b71: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:28.358263 16856 rebalancer_tool.cc:1084] tablet f65ab6dba1124573962e2da58aa1859c: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move ignored: Illegal state: tablet f65ab6dba1124573962e2da58aa1859c: replica c0fd1ebec9da48d4b0c1ae0f0ee96452 is already present
I20240503 08:15:28.358457 16856 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:15:29.967614 16856 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:30.259168 16856 rebalancer_tool.cc:1074] tablet 169bd9ea6772420abf757dd55cf44d31: '036223b0890d4634acfadf9596f39f80' -> '0d77c4b1f12049dea71b97e86be2b288' move scheduled
I20240503 08:15:31.178201 16856 rebalancer_tool.cc:1074] tablet c130eb7ea8a74f4b8d9b79e23af70067: '0f8702fad8ea4d22840dab11bb27f7f2' -> '0d77c4b1f12049dea71b97e86be2b288' move scheduled
I20240503 08:15:36.634702 16856 rebalancer_tool.cc:1136] tablet c130eb7ea8a74f4b8d9b79e23af70067: '0f8702fad8ea4d22840dab11bb27f7f2' -> '0d77c4b1f12049dea71b97e86be2b288' move completed: OK
I20240503 08:15:38.501332 16856 rebalancer_tool.cc:1136] tablet 169bd9ea6772420abf757dd55cf44d31: '036223b0890d4634acfadf9596f39f80' -> '0d77c4b1f12049dea71b97e86be2b288' move completed: OK
Illegal state: tablet server 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): unacceptable health status UNAVAILABLE

I20240503 08:15:41.058701 28384 ts_tablet_manager.cc:1888] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20240503 08:15:41.058822 28384 log.cc:1196] T 72a949e76ab5477680a4f9912e9672e9 P 0f8702fad8ea4d22840dab11bb27f7f2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/wals/72a949e76ab5477680a4f9912e9672e9
I20240503 08:15:41.059590 14334 catalog_manager.cc:4830] TS 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): tablet 72a949e76ab5477680a4f9912e9672e9 (table rebalance_test_table_7 [id=4e5e40744d3943e7b71fb29a47a92384]) successfully deleted
W20240503 08:15:41.156155 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:41.237311 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:41.238343 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:41.251070 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:41.258020 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:41.272871 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:41.307924 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:41.308094 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:41.417407 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:41.448407 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:41.466327 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:41.482304 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:41.484233 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:41.486407 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:41.524179 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:41.534709 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:41.566993 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:41.580420 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.626539 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.647644 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:41.656711 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.689396 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:41.697443 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:41.704670 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:41.711242 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.712188 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:41.721951 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:41.729826 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.741933 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:41.747277 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:41.748273 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:41.775465 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:41.824810 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:41.850718 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.861361 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.911474 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.914530 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.949714 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:41.981247 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:42.000301 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:42.075232 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:42.092929 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:42.102995 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:42.107698 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:42.116108 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:42.166671 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:42.167862 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:42.189253 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:42.193502 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:42.195077 16854 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 08:14:29.407184 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:29.415853 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:29.440515 16858 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:32.975944 16858 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:14:35.254606 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.254854 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.255160 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.255313 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.255434 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.255618 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:35.255725 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.255867 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.255975 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:35.256091 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.256259 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:35.256393 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:35.256500 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:35.256650 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:35.256753 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:35.668273 16858 rebalancer_tool.cc:1074] tablet 88404b9cdf574cb0bca1bb49348fb4cd: '0d77c4b1f12049dea71b97e86be2b288' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:36.017138 16858 rebalancer_tool.cc:1084] tablet dc9575f64eb741948e51ccf06f48228a: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 08:14:36.017360 16858 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:38.002734 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:38.002990 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.003096 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.003293 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.003446 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:38.003568 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:38.003669 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.003806 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.003918 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:38.004057 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.004215 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.004402 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:38.004516 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.004642 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:38.055662 16858 rebalancer_tool.cc:1074] tablet f65ab6dba1124573962e2da58aa1859c: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:14:38.485760 16858 rebalancer_tool.cc:1074] tablet 1c687520c0a349fd8438dacbd5011001: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:38.894492 16858 rebalancer_tool.cc:1074] tablet dd11460051ab40febc2fe6aff58b5028: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:39.620644 16858 rebalancer_tool.cc:1074] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:40.218384 16858 rebalancer_tool.cc:1074] tablet b19df3ae9f2b44a3818855f259962fff: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:40.462853 16858 rebalancer_tool.cc:1074] tablet 30b13cdf4b234f17872f5d1588915968: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:14:40.800063 16858 rebalancer_tool.cc:1074] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:41.232053 16858 rebalancer_tool.cc:1074] tablet 7aa1031b9e30478597da5ad3708bdff5: '0d77c4b1f12049dea71b97e86be2b288' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:41.407027 16858 rebalancer_tool.cc:1084] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move ignored: Illegal state: tablet 76a216295a8a4feaa2d3326035879073: replica 6835ad958e094ce4927135b942a2e8a8 is already present
I20240503 08:14:41.407352 16858 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:43.475431 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:43.554255 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:43.554889 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:43.555073 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:43.556893 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:43.557971 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:43.558764 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:43.558959 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:43.559213 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:43.559371 16858 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
I20240503 08:14:43.812536 16858 rebalancer_tool.cc:1074] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:14:44.099309 16858 rebalancer_tool.cc:1074] tablet dfa711e8c8564bc78846b16d68470cea: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:44.189580 16858 rebalancer_tool.cc:1074] tablet a87c5af03f4d47d38bbfbd225c7195d1: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:01.570071 16858 rebalancer_tool.cc:1136] tablet 88404b9cdf574cb0bca1bb49348fb4cd: '0d77c4b1f12049dea71b97e86be2b288' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:08.470748 16858 rebalancer_tool.cc:1136] tablet b19df3ae9f2b44a3818855f259962fff: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:08.562497 16858 rebalancer_tool.cc:1136] tablet dfa711e8c8564bc78846b16d68470cea: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:08.631043 16858 rebalancer_tool.cc:1136] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:08.688997 16858 rebalancer_tool.cc:1136] tablet a87c5af03f4d47d38bbfbd225c7195d1: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:08.737082 16858 rebalancer_tool.cc:1136] tablet 30b13cdf4b234f17872f5d1588915968: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:08.786263 16858 rebalancer_tool.cc:1136] tablet 7aa1031b9e30478597da5ad3708bdff5: '0d77c4b1f12049dea71b97e86be2b288' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:08.884581 16858 rebalancer_tool.cc:1136] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:08.926539 16858 rebalancer_tool.cc:1136] tablet f65ab6dba1124573962e2da58aa1859c: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:09.014077 16858 rebalancer_tool.cc:1136] tablet 1c687520c0a349fd8438dacbd5011001: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:09.115021 16858 rebalancer_tool.cc:1136] tablet dd11460051ab40febc2fe6aff58b5028: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:09.187152 16858 rebalancer_tool.cc:1136] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:09.219583 16858 rebalancer_tool.cc:1074] tablet c130eb7ea8a74f4b8d9b79e23af70067: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:09.307598 16858 rebalancer_tool.cc:1074] tablet 897a701b880641b08816f2d1a254ef03: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:09.531877 16858 rebalancer_tool.cc:1074] tablet 5c595bb19d6946b0b1f853064892e946: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:09.706548 16858 rebalancer_tool.cc:1074] tablet 6b677894b51f4dc58f157d8f4d632e9a: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:09.815984 16858 rebalancer_tool.cc:1074] tablet 12df78f752c34fe3885b235c4a19e325: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:09.969700 16858 rebalancer_tool.cc:1136] tablet 12df78f752c34fe3885b235c4a19e325: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:10.022892 16858 rebalancer_tool.cc:1136] tablet 6b677894b51f4dc58f157d8f4d632e9a: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:10.081256 16858 rebalancer_tool.cc:1136] tablet 897a701b880641b08816f2d1a254ef03: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:10.152418 16858 rebalancer_tool.cc:1136] tablet 5c595bb19d6946b0b1f853064892e946: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:10.220813 16858 rebalancer_tool.cc:1136] tablet c130eb7ea8a74f4b8d9b79e23af70067: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:10.323295 16858 rebalancer_tool.cc:1074] tablet 169bd9ea6772420abf757dd55cf44d31: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:10.587109 16858 rebalancer_tool.cc:1074] tablet 562e1ec5c34945298cf457758328057f: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:15:10.808948 16858 rebalancer_tool.cc:1074] tablet 4e86f4f3a01048e69f553c6557d79b71: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:10.960027 16858 rebalancer_tool.cc:1074] tablet 72a949e76ab5477680a4f9912e9672e9: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:11.048307 16858 rebalancer_tool.cc:1074] tablet f7a5221747794a13aa4ed763006ba291: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:19.706408 16858 rebalancer_tool.cc:1136] tablet 562e1ec5c34945298cf457758328057f: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:19.823984 16858 rebalancer_tool.cc:1136] tablet 169bd9ea6772420abf757dd55cf44d31: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:19.945765 16858 rebalancer_tool.cc:1074] tablet 55067e08fb694e90b6366d41cc962095: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:20.038028 16858 rebalancer_tool.cc:1136] tablet 55067e08fb694e90b6366d41cc962095: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:22.200222 16858 rebalancer_tool.cc:1136] tablet f7a5221747794a13aa4ed763006ba291: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:22.399271 16858 rebalancer_tool.cc:1136] tablet 4e86f4f3a01048e69f553c6557d79b71: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:24.979574 16858 rebalancer_tool.cc:1136] tablet 72a949e76ab5477680a4f9912e9672e9: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:27.759646 16858 rebalancer_tool.cc:1074] tablet f65ab6dba1124573962e2da58aa1859c: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move scheduled
I20240503 08:15:28.333319 16858 rebalancer_tool.cc:1074] tablet 5d16c2981213464cbefa44b6ca509182: '95860e22c9474e2d894f04deba8442cc' -> '129a2ac87d7f416db6d47d79bb1c2b5f' move scheduled
I20240503 08:15:28.779911 16858 rebalancer_tool.cc:1074] tablet 72a949e76ab5477680a4f9912e9672e9: '0f8702fad8ea4d22840dab11bb27f7f2' -> '129a2ac87d7f416db6d47d79bb1c2b5f' move scheduled
I20240503 08:15:29.631247 16858 rebalancer_tool.cc:1074] tablet 55067e08fb694e90b6366d41cc962095: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move scheduled
I20240503 08:15:37.856985 16858 rebalancer_tool.cc:1136] tablet 55067e08fb694e90b6366d41cc962095: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move completed: OK
W20240503 08:15:39.344503 28119 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:38.736073 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.135:46299 (local address 127.0.0.1:53478)
0503 08:15:38.753437 (+ 17364us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:38.753457 (+    20us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:38.753560 (+   103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:39.343031 (+589471us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":17246}
I20240503 08:15:39.413503 16858 rebalancer_tool.cc:1136] tablet 5d16c2981213464cbefa44b6ca509182: '95860e22c9474e2d894f04deba8442cc' -> '129a2ac87d7f416db6d47d79bb1c2b5f' move completed: OK
I20240503 08:15:39.594501 16858 rebalancer_tool.cc:1136] tablet f65ab6dba1124573962e2da58aa1859c: '6835ad958e094ce4927135b942a2e8a8' -> 'c0fd1ebec9da48d4b0c1ae0f0ee96452' move completed: OK
I20240503 08:15:41.132306 16858 rebalancer_tool.cc:1136] tablet 72a949e76ab5477680a4f9912e9672e9: '0f8702fad8ea4d22840dab11bb27f7f2' -> '129a2ac87d7f416db6d47d79bb1c2b5f' move completed: OK
Illegal state: tablet server 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): unacceptable health status UNAVAILABLE

W20240503 08:15:42.197484 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:42.220496 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:15:42.229547 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:15:42.384964 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:42.405632 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:42.412169 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:42.422753 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:42.441190 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:42.445541 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:42.512166 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:42.709764 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:42.714138 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:42.866036 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:42.871798 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:42.892603 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:42.912151 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:42.915321 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:42.932201 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:42.985627 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:43.125803 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:43.147688 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:43.155017 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:43.155869 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:43.163563 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:43.189157 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:43.198887 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:43.243165 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:43.371925 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:43.382272 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:43.391453 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:43.391587 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:43.408057 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:43.411444 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:43.434808 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:43.650628 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:43.732066 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:43.739691 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:43.822438 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:43.825783 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:43.827431 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:43.833433 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:43.837319 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:43.841949 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:43.856442 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:43.963172 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:43.977005 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:44.003755 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.040884 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:15:44.071910 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:44.083336 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:44.090126 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:44.095762 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:44.103448 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.109068 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:15:44.113337 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:44.115294 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:15:44.135871 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:44.174818 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:44.225400 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:44.318517 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.333365 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:15:44.334446 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:44.342615 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:44.346340 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240503 08:15:44.353055 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:44.354040 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:44.354892 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:44.359560 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:44.360291 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.427610 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:15:44.428881 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.429318 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:44.443014 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:15:44.538106 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.539306 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.540233 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.548005 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.571657 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.619149 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:44.647253 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.673110 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:15:44.754658 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:44.761349 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:44.777325 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:44.790352 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:44.807387 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:44.840759 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:44.841187 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:44.851276 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:15:44.941087 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:15:44.974977 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:45.018564 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:45.030763 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:45.031757 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:45.032361 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:45.049039 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:45.049340 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:45.070259 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:45.287186 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:45.288564 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:45.332850 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:45.343335 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:45.358036 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:45.369550 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:45.396457 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:45.556015 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:45.595294 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:45.771087 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:45.782038 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:45.801108 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:45.801505 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:45.809468 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:45.835753 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:45.864392 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:46.019114 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:46.037768 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:46.038869 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:46.044634 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:46.061712 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:46.062489 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:46.083698 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:46.097121 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:46.268260 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:46.278544 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:46.292119 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:46.297561 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:46.298127 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:46.299902 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:46.301286 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:46.369113 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.422892 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.453397 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:46.462000 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.472720 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.501832 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:46.533912 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.546001 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:46.575713 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.580473 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.665701 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:46.711131 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:46.727660 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:46.735091 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:46.741997 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:46.776304 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:46.785343 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:46.787158 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:46.816565 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.862798 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.922232 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:46.945372 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:46.994655 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:47.035982 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:47.036146 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:47.036906 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:47.037143 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:47.052776 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:47.066833 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:47.074729 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:47.078204 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:47.080927 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:47.126909 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:47.167019 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:47.244508 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:47.252275 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:47.257225 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:47.261794 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:47.272619 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:15:47.278831 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:47.283682 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:47.304253 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:47.383420 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:47.505556 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:15:47.625571 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:47.640681 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:47.670850 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:47.672333 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:47.677605 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:47.707226 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:47.723238 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:47.727062 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:47.929454 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:47.941056 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:47.944674 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:47.958312 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:47.970243 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:47.977198 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:47.997646 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:48.055655 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:48.069640 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:48.188105 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:48.199383 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:48.199962 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:48.200331 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:48.216984 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:48.231294 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:48.551381 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:48.554208 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:48.556265 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:48.562903 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:48.623436 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:48.629494 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:48.642498 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:48.654196 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:48.662101 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:48.666066 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:48.790395 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:48.838733 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:48.884495 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:15:48.890429 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:48.891660 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:48.904721 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:48.946624 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:48.947033 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:48.956979 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:48.957424 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:48.976994 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:48.994935 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:49.000082 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.027246 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.037039 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.089954 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.105778 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:49.147626 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:49.152312 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:49.165025 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:49.169173 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:49.201714 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:49.219832 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:49.256104 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.293457 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.311264 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.326745 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:49.359885 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:15:49.369128 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:49.431459 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:15:49.432425 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.493500 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.520949 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.522758 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.593907 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:15:49.615870 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:49.627561 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:49.634449 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:49.635193 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:49.648418 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:49.653369 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:49.679273 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:49.769610 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.779810 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:49.793815 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:49.816118 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:49.825482 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:15:49.845955 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:49.849396 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:49.859719 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:49.869343 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:49.870747 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:49.871541 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:50.099292 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:50.115406 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:50.125248 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:50.132355 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:50.160842 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:50.165810 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:50.165957 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:50.193589 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:15:50.245433 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:15:50.291436 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:50.387300 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:50.571647 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:50.586560 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:50.588790 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:50.589602 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:50.604694 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:50.611050 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:50.685828 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:50.714697 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:50.823302 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:50.831300 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:50.852794 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:50.862268 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:50.867959 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:50.885524 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:50.947826 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:51.067158 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.075343 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:51.105247 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:51.119046 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:51.123728 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:51.131290 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:51.142676 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:51.147459 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:51.154436 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:51.265532 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.320611 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.324523 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:51.340891 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.370422 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.463761 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.505795 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:51.570355 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:51.575482 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:51.584738 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:51.592288 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:51.593122 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:51.600574 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:51.612900 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:51.615082 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:51.620689 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:51.647521 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:51.713366 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:51.746310 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:51.814999 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:51.815856 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:51.826967 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:51.828092 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:51.849699 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:51.856011 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:51.897791 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.926735 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.932406 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.933627 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:51.968086 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:52.000043 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:52.086256 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:52.108311 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:52.108919 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:52.111584 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:52.113148 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:52.119434 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:52.138106 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:52.140043 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:52.249697 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:52.299273 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:52.393291 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:52.435128 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:15:52.519518 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:52.550065 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:52.553511 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:52.569042 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:52.591581 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:52.594897 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:52.624001 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:52.664249 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:15:52.772357 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:52.784471 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:52.785277 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:52.793246 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:52.795528 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:52.816174 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:52.832455 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:52.854406 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:53.055619 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:53.059957 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:53.062335 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:53.082726 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:53.095741 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:53.097822 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:53.215277 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:53.339150 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:53.415586 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:53.511564 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:53.517009 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:53.524142 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:53.526968 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:53.528368 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:53.536298 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:53.643385 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:53.666229 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:15:53.759480 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:53.775749 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:53.786484 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:53.786896 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:15:53.793790 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:53.798800 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:53.822969 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:53.866427 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:53.933805 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:53.949290 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:53.963876 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:54.019973 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:54.035238 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:54.041062 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:54.043076 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:54.047022 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:54.053336 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:54.064392 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:54.065578 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.072031 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.080946 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.168278 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.273998 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.297577 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.313690 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.403435 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:54.407212 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:15:54.408087 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.414255 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:54.457892 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:15:54.458734 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:54.463543 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:15:54.464385 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:15:54.472100 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:54.473472 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:54.487831 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:54.543310 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:54.543673 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:54.564622 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.621532 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.626644 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:54.638383 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:15:54.694677 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:15:54.695578 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:15:54.740592 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:54.754105 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:54.755683 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:54.761116 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:54.776566 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:54.783794 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:54.838891 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:54.930503 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:54.995615 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:55.004155 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:55.024425 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:55.031047 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:55.033221 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:55.040241 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:55.045537 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:15:55.069365 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:55.387228 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:55.446192 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:55.455788 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:15:55.467240 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:55.468030 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:55.471814 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:55.500744 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:55.522408 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:55.532599 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:55.638418 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:55.678463 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:55.694715 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:55.712782 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:55.713045 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:55.773550 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:55.778376 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:55.922514 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:55.957795 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:55.980268 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:56.000007 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:56.000310 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:56.009271 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:56.010704 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:56.060441 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:56.092458 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.130694 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.224828 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:56.231864 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.341862 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.349906 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.384261 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:15:56.426079 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:56.431466 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.438843 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:56.451545 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:56.461905 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:56.469519 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:56.492270 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:56.511780 14817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.131:45663, user_credentials={real_user=slave}} blocked reactor thread for 44109us
I20240503 08:15:56.515990 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:56.631336 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:56.659583 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.666967 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:15:56.692119 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:56.703298 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:56.717558 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:56.737115 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:56.746199 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:56.765149 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:56.829051 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:56.850468 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:56.855183 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.887945 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:56.924417 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:15:56.971259 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:56.990406 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:57.004601 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:15:57.006417 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:57.007095 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:57.019950 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:57.039697 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:15:57.043061 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:57.082728 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:57.119323 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:57.127126 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:57.169914 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:57.182063 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:57.255208 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:57.304351 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:15:57.321597 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:57.436828 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:57.439810 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:15:57.462488 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:57.471099 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:57.478448 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:57.519466 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:57.519794 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:57.635313 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:57.655050 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:57.700388 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:57.744102 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:57.744676 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:57.771433 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:57.806701 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:57.809453 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:57.830724 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:57.998044 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:58.007067 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:58.032061 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:58.035440 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:58.060278 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:58.063473 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:58.123034 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:58.157789 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:58.273322 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:58.442015 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:58.442966 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:58.446275 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:58.459642 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:58.476619 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:58.481218 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:58.584910 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:58.673280 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:58.685628 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:58.694273 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:58.698474 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:58.699643 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:58.727030 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:58.743724 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:58.744982 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:58.765938 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:58.784128 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:58.836326 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:58.842621 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:15:58.934517 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:15:58.936448 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:58.946432 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:15:58.969261 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:58.972225 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:58.973649 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:59.016306 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:59.041112 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:15:59.051182 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:59.118808 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.178622 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.243798 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:59.243855 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:59.256091 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.277089 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.295428 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:59.407634 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:59.424878 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:59.425815 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:59.441720 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:59.442044 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:59.444751 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:59.464095 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.487656 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:15:59.488624 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.504845 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.593995 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.598623 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:15:59.599596 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:59.622670 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:59.656098 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:59.671773 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:59.674801 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:59.679996 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:59.684980 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:15:59.708873 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:59.728596 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:15:59.740481 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:15:59.767079 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:15:59.768008 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:15:59.804996 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.825114 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:15:59.856536 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:15:59.914956 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:15:59.917440 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:15:59.925037 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:15:59.946934 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:15:59.975136 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:15:59.987466 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:00.185120 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:00.303539 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:00.304047 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:00.360841 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:00.367923 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:00.395644 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:00.404268 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:00.419698 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:00.433137 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:00.462198 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:16:00.598107 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:00.601478 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:00.637121 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:00.643170 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:00.676412 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:00.679052 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:00.716447 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:00.850292 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:00.872871 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:00.876371 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:00.880903 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:00.881668 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:00.883555 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:00.912969 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:00.953486 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:01.107450 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.118206 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:01.191663 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.237700 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.295607 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:01.312222 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:16:01.348817 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:01.355819 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:01.374285 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:01.385681 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:01.390242 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:01.402334 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:01.402582 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:16:01.404700 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:01.520637 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.525439 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.549103 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:01.593700 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:01.604866 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:01.622406 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:01.626819 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:01.652729 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.655102 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.667451 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:16:01.668784 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:01.675483 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:01.703020 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:01.783669 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:01.802816 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:01.874864 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:01.902040 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:01.909613 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:01.933952 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:01.941713 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:16:01.962312 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:01.991448 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:02.069872 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.077900 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.084658 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:02.086891 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.189522 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.249673 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.256182 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.271526 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:16:02.352252 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:02.403882 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:02.407166 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:02.411300 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:02.428462 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:02.433456 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:02.437283 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:02.458375 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:02.525725 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:16:02.649223 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:02.658632 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:02.662740 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:02.681715 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:02.688464 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:02.708730 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:02.744289 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:02.782377 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:02.856730 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:02.950407 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:02.963141 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:02.969223 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:02.980188 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:03.017561 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:03.074812 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:03.093092 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:03.347779 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:03.446136 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:03.457006 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:03.472822 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:03.483211 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:03.484046 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:03.488317 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:03.503283 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:03.591267 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:03.597847 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:03.619693 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:16:03.704015 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:03.723639 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:03.730118 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:03.759380 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:03.779070 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:03.791275 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:03.793098 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:03.799644 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:03.868081 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:03.964893 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:16:03.997236 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:04.005504 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:04.006738 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:04.008607 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:04.013016 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:04.021240 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:04.032701 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.060428 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:04.110069 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.127148 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.162369 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.167910 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.168937 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:04.188511 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.301244 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.325241 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:04.419340 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:04.422521 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:04.442670 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:04.444360 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:04.471913 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:04.499148 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:04.502584 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:04.574505 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:16:04.575345 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.620393 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:16:04.621253 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:04.627028 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.650290 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.668831 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:16:04.712663 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:04.715978 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:04.728688 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.739439 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:16:04.745919 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:04.775754 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:04.786562 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:04.802695 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:04.827342 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:16:04.847767 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:04.850780 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:04.895509 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:04.977680 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:16:04.995745 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:05.017549 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:05.018522 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:05.036756 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:05.071081 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:05.076822 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:05.244629 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:05.396528 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:05.417335 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:05.441517 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:05.468067 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:05.473487 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:05.476703 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:05.506904 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:05.507000 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:05.645293 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:16:05.701010 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:05.724546 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:05.742280 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:05.742985 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:05.752127 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:05.756747 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:05.779529 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:05.864567 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:06.000414 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:06.014066 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:06.016103 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:06.024444 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:06.033046 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:06.042529 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:06.063552 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:06.086366 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:06.188838 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:06.233795 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:06.378690 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:06.399492 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:06.433243 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:06.451392 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:06.451452 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:06.462944 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:06.477802 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:06.481235 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:06.500188 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:06.505499 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:06.519799 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:06.540109 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:06.562908 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:06.586117 14338 maintenance_manager.cc:419] P 309eed6422ad4e71bcf73f518dc42423: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033391
W20240503 08:16:06.614105 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:06.637035 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:06.642200 14337 maintenance_manager.cc:643] P 309eed6422ad4e71bcf73f518dc42423: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.055s	user 0.038s	sys 0.013s Metrics: {"bytes_written":20123,"cfile_init":1,"dirs.queue_time_us":401,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":897,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":512,"threads_started":2,"wal-append.queue_time_us":344}
I20240503 08:16:06.643623 14338 maintenance_manager.cc:419] P 309eed6422ad4e71bcf73f518dc42423: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9242 bytes on disk
I20240503 08:16:06.644547 14337 maintenance_manager.cc:643] P 309eed6422ad4e71bcf73f518dc42423: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 08:16:06.702111 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:06.712750 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:06.714463 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:06.732141 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:06.746088 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:06.770663 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:06.788475 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:06.790006 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:06.847090 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:06.966558 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:06.986555 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:06.988253 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:06.998132 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:07.014318 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:07.026703 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:07.030859 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:07.070549 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:07.107365 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:07.157804 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:07.175545 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:07.176107 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:07.183243 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:07.239553 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:07.318734 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:07.339814 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:07.435120 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:07.435097 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:16:07.444306 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:07.463582 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:07.473781 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:07.492060 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:07.527221 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:07.557408 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:07.606388 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:07.609527 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:16:07.676033 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:07.723711 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:07.744185 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:07.751080 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:07.752102 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:07.780121 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:07.826462 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:08.016117 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:08.020133 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:08.022343 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:08.030825 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:08.035446 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:08.040807 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:08.046412 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:08.074450 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:08.264122 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:08.438355 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:08.452834 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:08.466416 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:08.479949 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:08.481287 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:08.507805 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:08.523604 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:08.619642 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:08.661393 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:08.712188 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:08.714547 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:08.714599 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:08.732882 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:08.735996 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:08.747260 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:08.751746 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:08.764168 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:08.909838 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:08.965173 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:08.983224 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:08.985301 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:08.998122 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:09.013505 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.027660 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:09.036883 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:09.037494 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:09.044706 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:09.081877 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.135605 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:09.216140 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.255110 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.255115 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.264467 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.336721 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.378774 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:09.420491 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:09.430613 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:09.432711 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:09.450915 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:09.452298 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:09.455765 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:09.459297 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:09.463441 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:09.506994 14425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 49882.4us
W20240503 08:16:09.509407 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:09.514654 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.607151 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.664387 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:09.696622 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:09.697196 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240503 08:16:09.699318 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:09.702723 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:09.706794 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:09.716569 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:09.717522 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:09.733186 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:16:09.737202 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.849292 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.870149 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:09.919190 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:09.928272 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:16:09.938602 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:09.938771 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:09.966470 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:09.980556 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:09.981429 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:09.982510 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:09.987723 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:09.997251 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:10.032996 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:10.170470 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:16:10.244820 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:10.384801 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:10.388631 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:10.391498 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:10.394059 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:10.411151 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:10.422286 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:10.452428 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:10.563342 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:10.627997 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:10.632292 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:10.634600 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:10.656193 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:10.670969 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:10.685690 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:10.851645 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:10.860257 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:10.877172 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:16:10.898361 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:10.915546 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:10.925112 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:10.933384 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:10.956128 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:10.983914 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:11.042035 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.109297 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.185181 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:11.210391 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.260054 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:11.342023 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:11.380548 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:11.394937 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:11.395632 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:11.401638 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:11.404417 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:11.425560 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:11.442457 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:11.477847 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:11.581993 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:11.678258 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:11.679689 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:11.699164 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:11.702034 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:11.703445 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:11.737370 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.748940 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:11.751844 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:11.763387 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:11.790704 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:11.799409 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.799472 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.807154 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:11.891263 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:12.044706 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:12.045639 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:12.046072 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:12.050873 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:12.056350 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:12.070325 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:12.081260 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:12.107311 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:12.113137 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:12.130807 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:12.202247 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:12.252453 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:12.271906 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:12.352659 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:12.483808 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:12.494035 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:12.499825 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:12.521030 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:12.532441 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:12.583030 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:16:12.608525 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:12.611660 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:12.622198 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:12.633162 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:16:12.706768 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:12.802727 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:12.815490 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:12.823300 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:12.823726 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:12.833626 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:12.846395 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:12.864252 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:13.024024 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:13.100975 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:13.102670 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:13.107873 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:13.109825 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:13.130118 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:13.132473 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:13.191774 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:13.266888 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:13.281236 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:13.418718 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:13.467500 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:13.539412 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:13.549060 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:13.554888 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:13.579854 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:13.585665 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:13.594864 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:13.717304 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:13.718214 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:13.787998 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:13.790115 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:13.806680 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:13.809176 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:13.821314 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:13.831557 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:13.851575 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:13.877275 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:13.918612 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:13.972064 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:13.978900 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:14.039031 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:14.069949 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:14.070359 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:14.074143 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:14.091485 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:14.109411 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:14.113303 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:14.162876 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:14.223202 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:14.237234 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:14.254945 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:14.275909 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:14.333030 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:14.449151 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:14.495901 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:14.509843 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:14.519001 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:14.531706 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:14.542418 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:14.548383 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:14.560220 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:14.568617 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:14.670410 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:14.713953 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:14.763005 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(dc9575f64eb741948e51ccf06f48228a): perf score=0.033365
W20240503 08:16:14.768216 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:16:14.776504 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:14.776577 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:14.779166 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:14.791774 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 124 similar messages]
I20240503 08:16:14.797446 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:14.803644 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(6b677894b51f4dc58f157d8f4d632e9a): perf score=0.033378
W20240503 08:16:14.827512 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:14.840719 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:14.845777 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:16:14.856396 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(f65ab6dba1124573962e2da58aa1859c): perf score=0.033399
W20240503 08:16:14.861472 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 70983.1us
W20240503 08:16:14.864668 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:14.869776 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:14.879063 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:14.897728 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:14.987250 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:15.051544 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:15.061625 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:16:15.089874 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(6b677894b51f4dc58f157d8f4d632e9a) complete. Timing: real 0.285s	user 0.073s	sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":15139,"dirs.queue_time_us":11440,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1044,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1048,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42114,"threads_started":3,"wal-append.queue_time_us":10447}
I20240503 08:16:15.092000 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(6b677894b51f4dc58f157d8f4d632e9a): 4923 bytes on disk
I20240503 08:16:15.092832 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(6b677894b51f4dc58f157d8f4d632e9a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 08:16:15.118197 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(dc9575f64eb741948e51ccf06f48228a) complete. Timing: real 0.354s	user 0.066s	sys 0.026s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":13508,"dirs.queue_time_us":14352,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1020,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1309,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":45126,"threads_started":3,"wal-append.queue_time_us":3468}
I20240503 08:16:15.122179 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(dc9575f64eb741948e51ccf06f48228a): 4836 bytes on disk
I20240503 08:16:15.123709 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(dc9575f64eb741948e51ccf06f48228a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":519,"lbm_reads_lt_1ms":4}
I20240503 08:16:15.160568 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(f65ab6dba1124573962e2da58aa1859c) complete. Timing: real 0.303s	user 0.095s	sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":23979,"dirs.queue_time_us":10337,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":5718,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3480,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":44600,"threads_started":3,"wal-append.queue_time_us":8114}
I20240503 08:16:15.162505 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(f65ab6dba1124573962e2da58aa1859c): 4583 bytes on disk
I20240503 08:16:15.165416 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(f65ab6dba1124573962e2da58aa1859c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 08:16:15.232039 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:15.312101 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:16:15.346385 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:15.467402 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:15.551723 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:15.578018 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:15.587962 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:15.604224 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:15.605973 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:15.621265 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:15.658012 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:15.774124 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:15.800423 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:15.847046 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(dd11460051ab40febc2fe6aff58b5028): perf score=0.033337
I20240503 08:16:15.877627 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(169bd9ea6772420abf757dd55cf44d31): perf score=0.033347
I20240503 08:16:15.924880 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(dd11460051ab40febc2fe6aff58b5028) complete. Timing: real 0.077s	user 0.048s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18304,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1542,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1038,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":25163,"threads_started":2,"wal-append.queue_time_us":6867}
I20240503 08:16:15.926544 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(dd11460051ab40febc2fe6aff58b5028): 4838 bytes on disk
I20240503 08:16:15.927325 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(dd11460051ab40febc2fe6aff58b5028) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240503 08:16:15.986076 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(169bd9ea6772420abf757dd55cf44d31) complete. Timing: real 0.108s	user 0.045s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":28988,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":5984,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":838,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6595,"threads_started":2,"wal-append.queue_time_us":400}
I20240503 08:16:15.987516 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(55067e08fb694e90b6366d41cc962095): perf score=0.033372
I20240503 08:16:16.021523 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(55067e08fb694e90b6366d41cc962095) complete. Timing: real 0.034s	user 0.020s	sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":237,"threads_started":1,"wal-append.queue_time_us":340}
I20240503 08:16:16.022804 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(169bd9ea6772420abf757dd55cf44d31): 4923 bytes on disk
I20240503 08:16:16.023465 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(169bd9ea6772420abf757dd55cf44d31) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240503 08:16:16.024519 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(55067e08fb694e90b6366d41cc962095): 4583 bytes on disk
I20240503 08:16:16.025054 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(55067e08fb694e90b6366d41cc962095) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 08:16:16.030825 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:16.036309 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:16.154721 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:16.156587 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:16.179725 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:16.204949 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:16.205061 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:16.206952 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:16.217212 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:16.228251 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:16.271760 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:16.365557 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:16.418351 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:16.421509 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:16.428755 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:16.434873 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:16.453473 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:16.466951 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:16.471487 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:16.478988 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:16.495085 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:16.539227 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:16.572626 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:16.641961 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:16.686321 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:16.689770 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:16.706739 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:16.721462 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:16.732837 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:16.744614 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:16.758868 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:16.776960 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:16.777583 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:16.813788 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:16.880403 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:16.897593 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:16.925001 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(3fea213e5a12452f8b894b9cdb94f9ae): perf score=0.033360
I20240503 08:16:16.934096 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(30b13cdf4b234f17872f5d1588915968): perf score=0.033357
I20240503 08:16:16.987017 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(30b13cdf4b234f17872f5d1588915968) complete. Timing: real 0.053s	user 0.034s	sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6888,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":11745,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":901,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7192,"threads_started":2,"wal-append.queue_time_us":294}
I20240503 08:16:16.987286 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(3fea213e5a12452f8b894b9cdb94f9ae) complete. Timing: real 0.062s	user 0.042s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15123,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":2977,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":791,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15226,"threads_started":2,"wal-append.queue_time_us":382}
I20240503 08:16:16.988301 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(30b13cdf4b234f17872f5d1588915968): 4838 bytes on disk
I20240503 08:16:16.988651 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(3fea213e5a12452f8b894b9cdb94f9ae): 4923 bytes on disk
I20240503 08:16:16.988782 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(30b13cdf4b234f17872f5d1588915968) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:16:16.989169 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(3fea213e5a12452f8b894b9cdb94f9ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 08:16:17.031077 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(12df78f752c34fe3885b235c4a19e325): perf score=0.033383
W20240503 08:16:17.033519 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:17.075197 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(12df78f752c34fe3885b235c4a19e325) complete. Timing: real 0.044s	user 0.031s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":341,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":461,"threads_started":2,"wal-append.queue_time_us":327}
I20240503 08:16:17.076450 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(12df78f752c34fe3885b235c4a19e325): 4583 bytes on disk
I20240503 08:16:17.077126 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(12df78f752c34fe3885b235c4a19e325) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240503 08:16:17.080520 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(12df78f752c34fe3885b235c4a19e325): perf score=0.033403
W20240503 08:16:17.104897 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:17.117767 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:17.167245 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(12df78f752c34fe3885b235c4a19e325) complete. Timing: real 0.086s	user 0.053s	sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":353,"dirs.queue_time_us":11015,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":922,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11729,"threads_started":3,"wal-append.queue_time_us":394}
I20240503 08:16:17.168574 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(12df78f752c34fe3885b235c4a19e325): 4583 bytes on disk
I20240503 08:16:17.169258 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(12df78f752c34fe3885b235c4a19e325) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 08:16:17.219241 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:17.245085 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:17.251843 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:17.255491 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:17.258414 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:17.285559 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:17.292135 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:17.306270 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:17.313179 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:17.325803 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:17.472364 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:17.524675 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:17.533437 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:17.563176 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:17.570396 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:17.587057 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:17.610698 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:17.630817 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:17.639400 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:16:17.645365 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:17.718675 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:17.770210 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:17.852684 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:16:17.854070 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:17.854825 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:17.855993 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:17.867015 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:17.888360 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:17.894395 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:17.970669 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:17.993428 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(7cc0fe7d81e149d5b82fabfdc11d66b4): perf score=0.033358
I20240503 08:16:17.995060 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(7cc0fe7d81e149d5b82fabfdc11d66b4): perf score=0.033357
I20240503 08:16:18.045924 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(7cc0fe7d81e149d5b82fabfdc11d66b4) complete. Timing: real 0.052s	user 0.038s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":674,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1189,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":880,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":816,"threads_started":2,"wal-append.queue_time_us":385}
I20240503 08:16:18.047276 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(7cc0fe7d81e149d5b82fabfdc11d66b4): 4923 bytes on disk
I20240503 08:16:18.047789 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(7cc0fe7d81e149d5b82fabfdc11d66b4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 08:16:18.048794 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(a87c5af03f4d47d38bbfbd225c7195d1): perf score=0.033367
I20240503 08:16:18.068583 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(7cc0fe7d81e149d5b82fabfdc11d66b4) complete. Timing: real 0.073s	user 0.058s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":345,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1617,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":373}
I20240503 08:16:18.069818 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(7cc0fe7d81e149d5b82fabfdc11d66b4): 4923 bytes on disk
I20240503 08:16:18.070473 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(7cc0fe7d81e149d5b82fabfdc11d66b4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240503 08:16:18.081797 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(0780fb979eee4bae98a014d9721c9441): perf score=0.033372
I20240503 08:16:18.084461 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(a87c5af03f4d47d38bbfbd225c7195d1) complete. Timing: real 0.035s	user 0.033s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1358,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":680,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":213,"threads_started":1,"wal-append.queue_time_us":322}
I20240503 08:16:18.086123 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(a87c5af03f4d47d38bbfbd225c7195d1): 4584 bytes on disk
I20240503 08:16:18.086697 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(a87c5af03f4d47d38bbfbd225c7195d1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:16:18.128670 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(0780fb979eee4bae98a014d9721c9441) complete. Timing: real 0.047s	user 0.040s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":358,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1587,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":456,"threads_started":2,"wal-append.queue_time_us":322}
I20240503 08:16:18.129747 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(0780fb979eee4bae98a014d9721c9441): 4838 bytes on disk
I20240503 08:16:18.130297 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(0780fb979eee4bae98a014d9721c9441) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240503 08:16:18.140955 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:18.157701 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:18.280884 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:18.303848 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:18.304982 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:18.315382 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:18.319079 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:18.380406 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:18.390483 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:18.447162 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:18.555429 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:18.588611 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:18.592885 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:18.602131 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:18.604485 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:18.604496 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:18.630635 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:18.652091 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:18.689575 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:18.778367 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:18.799093 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:18.801569 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:18.808153 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:18.811055 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:18.836608 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:18.839967 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:18.874986 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:18.880218 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:18.890292 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:18.984690 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:19.011281 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:19.075448 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(b19df3ae9f2b44a3818855f259962fff): perf score=0.033360
I20240503 08:16:19.091784 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(1c687520c0a349fd8438dacbd5011001): perf score=0.033352
I20240503 08:16:19.113426 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(b19df3ae9f2b44a3818855f259962fff) complete. Timing: real 0.038s	user 0.014s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":367,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":477,"threads_started":2,"wal-append.queue_time_us":314}
I20240503 08:16:19.115021 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(b19df3ae9f2b44a3818855f259962fff): 4583 bytes on disk
I20240503 08:16:19.115628 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(b19df3ae9f2b44a3818855f259962fff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 08:16:19.136268 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(b19df3ae9f2b44a3818855f259962fff): perf score=0.033384
I20240503 08:16:19.137383 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(1c687520c0a349fd8438dacbd5011001) complete. Timing: real 0.045s	user 0.036s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":327,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1504,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":449,"threads_started":2,"wal-append.queue_time_us":381}
I20240503 08:16:19.139207 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(1c687520c0a349fd8438dacbd5011001): 4838 bytes on disk
W20240503 08:16:19.139710 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:19.140146 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(1c687520c0a349fd8438dacbd5011001) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 08:16:19.157219 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:19.176728 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(1d6c920958214148b3c3d5ac995e99e2): perf score=0.033388
I20240503 08:16:19.181246 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(b19df3ae9f2b44a3818855f259962fff) complete. Timing: real 0.045s	user 0.038s	sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1970,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":828,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":523,"threads_started":2,"wal-append.queue_time_us":396}
I20240503 08:16:19.182564 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(b19df3ae9f2b44a3818855f259962fff): 4583 bytes on disk
I20240503 08:16:19.183145 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(b19df3ae9f2b44a3818855f259962fff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240503 08:16:19.221956 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:19.239784 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(1d6c920958214148b3c3d5ac995e99e2) complete. Timing: real 0.063s	user 0.041s	sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":310,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1718,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":734,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6692,"threads_started":2,"wal-append.queue_time_us":6362}
I20240503 08:16:19.241257 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(1d6c920958214148b3c3d5ac995e99e2): 4923 bytes on disk
I20240503 08:16:19.242568 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(1d6c920958214148b3c3d5ac995e99e2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240503 08:16:19.255024 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:19.276767 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:19.283174 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:19.294386 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:19.311793 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:19.326627 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:19.337989 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:19.349061 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:19.369697 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:19.483690 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:19.513105 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:19.531958 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:19.532356 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:19.543120 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:19.545416 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:19.566491 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:19.581049 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:19.651854 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:19.680810 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:19.695216 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:19.738111 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:19.763859 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:19.775785 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:19.778889 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:19.792634 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:19.793437 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240503 08:16:19.807390 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:19.809119 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:19.811058 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:19.845381 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:19.989964 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:20.040163 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:16:20.041129 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:16:20.069978 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:20.094332 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:20.187381 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(7aa1031b9e30478597da5ad3708bdff5): perf score=0.033351
W20240503 08:16:20.210865 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:20.234341 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:20.239730 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:20.247749 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(7aa1031b9e30478597da5ad3708bdff5): perf score=0.033360
I20240503 08:16:20.261358 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:20.301601 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:20.305269 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:20.305519 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:20.344345 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(7aa1031b9e30478597da5ad3708bdff5) complete. Timing: real 0.096s	user 0.038s	sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6412,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1530,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15704,"threads_started":2,"wal-append.queue_time_us":9508}
I20240503 08:16:20.349025 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(7aa1031b9e30478597da5ad3708bdff5) complete. Timing: real 0.161s	user 0.037s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1872,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":809,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21269,"threads_started":2,"wal-append.queue_time_us":15008}
I20240503 08:16:20.350380 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(7aa1031b9e30478597da5ad3708bdff5): 4923 bytes on disk
I20240503 08:16:20.351459 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(7aa1031b9e30478597da5ad3708bdff5): 4923 bytes on disk
I20240503 08:16:20.352995 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(7aa1031b9e30478597da5ad3708bdff5) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":385,"lbm_reads_lt_1ms":4}
I20240503 08:16:20.353677 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:20.354933 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(7aa1031b9e30478597da5ad3708bdff5) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:16:20.357245 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:20.372099 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(562e1ec5c34945298cf457758328057f): perf score=0.033390
I20240503 08:16:20.396245 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(5c595bb19d6946b0b1f853064892e946): perf score=0.033395
W20240503 08:16:20.397405 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:16:20.429792 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(562e1ec5c34945298cf457758328057f) complete. Timing: real 0.057s	user 0.039s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6254,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1536,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":898,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":6016,"thread_start_us":7249,"threads_started":2,"wal-append.queue_time_us":1099}
I20240503 08:16:20.431241 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(562e1ec5c34945298cf457758328057f): 4583 bytes on disk
I20240503 08:16:20.431901 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(562e1ec5c34945298cf457758328057f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:16:20.458973 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(5c595bb19d6946b0b1f853064892e946) complete. Timing: real 0.062s	user 0.054s	sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":318,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":886,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":906,"threads_started":2,"wal-append.queue_time_us":759}
I20240503 08:16:20.460275 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(5c595bb19d6946b0b1f853064892e946): 4838 bytes on disk
I20240503 08:16:20.460930 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(5c595bb19d6946b0b1f853064892e946) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240503 08:16:20.601176 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:20.603976 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:20.607751 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:20.626861 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:20.630558 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:20.633975 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:20.649513 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:20.653275 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:20.753670 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:20.823098 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:20.870780 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:20.882835 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:20.900127 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:20.904140 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:20.926647 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:20.941715 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:20.960618 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:20.964192 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:21.227890 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:21.228762 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:21.296487 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:21.360131 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:21.372190 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:21.388465 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:21.394785 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:21.395882 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:21.414973 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:21.437753 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(76a216295a8a4feaa2d3326035879073): perf score=0.033355
I20240503 08:16:21.446257 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:21.453981 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:21.467077 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:21.538252 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(76a216295a8a4feaa2d3326035879073) complete. Timing: real 0.100s	user 0.066s	sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1486,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":12690,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1610,"threads_started":2,"wal-append.queue_time_us":342}
I20240503 08:16:21.539541 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(76a216295a8a4feaa2d3326035879073): 4836 bytes on disk
I20240503 08:16:21.540100 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(76a216295a8a4feaa2d3326035879073) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:16:21.609833 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(897a701b880641b08816f2d1a254ef03): perf score=0.033395
I20240503 08:16:21.612610 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(5d16c2981213464cbefa44b6ca509182): perf score=0.033393
W20240503 08:16:21.620059 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:21.621104 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:21.672168 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(897a701b880641b08816f2d1a254ef03) complete. Timing: real 0.062s	user 0.037s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3548,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":2048,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":937,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5712,"threads_started":2,"wal-append.queue_time_us":1943}
I20240503 08:16:21.674414 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(897a701b880641b08816f2d1a254ef03): 4584 bytes on disk
W20240503 08:16:21.679044 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:21.684638 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(897a701b880641b08816f2d1a254ef03) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
W20240503 08:16:21.698488 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:21.701480 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(5d16c2981213464cbefa44b6ca509182) complete. Timing: real 0.087s	user 0.052s	sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6380,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":5793,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":929,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8742,"threads_started":2,"wal-append.queue_time_us":2338}
I20240503 08:16:21.702682 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(5d16c2981213464cbefa44b6ca509182): 4924 bytes on disk
I20240503 08:16:21.703400 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(5d16c2981213464cbefa44b6ca509182) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 08:16:21.727761 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:21.734390 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:21.746806 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:21.748909 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:21.749771 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:21.774057 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:21.776383 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:21.810776 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:21.888687 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:21.918232 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:21.956126 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:21.981292 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:22.015334 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:22.017817 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:22.026257 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:22.029620 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:22.030709 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:22.030707 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:22.110670 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:22.136900 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:22.140966 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:22.141202 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:22.272617 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:22.411684 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:22.432377 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:22.442304 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:22.449317 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:22.449337 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:22.476369 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:22.477074 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:22.534933 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:22.565299 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:16:22.631549 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:22.672588 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:22.682487 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:22.697366 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:22.706099 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:22.715206 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(72a949e76ab5477680a4f9912e9672e9): perf score=0.033333
W20240503 08:16:22.722335 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:22.722646 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:22.723764 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:22.756759 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:22.795073 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(72a949e76ab5477680a4f9912e9672e9) complete. Timing: real 0.080s	user 0.042s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7544,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":5422,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2692,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7918,"threads_started":2,"wal-append.queue_time_us":344}
I20240503 08:16:22.796427 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(72a949e76ab5477680a4f9912e9672e9): 4583 bytes on disk
I20240503 08:16:22.796988 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(72a949e76ab5477680a4f9912e9672e9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 08:16:22.942344 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(f19b7ef418974b06bdcc7125e8cf732d): perf score=0.033390
I20240503 08:16:22.962446 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:22.964897 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:22.975461 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:16:22.983619 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:23.013264 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:23.029263 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:23.034854 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:23.105845 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(f19b7ef418974b06bdcc7125e8cf732d) complete. Timing: real 0.163s	user 0.033s	sys 0.027s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2519,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1327,"drs_written":1,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12796,"threads_started":2,"wal-append.queue_time_us":7066}
I20240503 08:16:23.107198 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(f19b7ef418974b06bdcc7125e8cf732d): 4838 bytes on disk
I20240503 08:16:23.107705 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(f19b7ef418974b06bdcc7125e8cf732d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 08:16:23.110509 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:23.111024 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:23.165455 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:23.346220 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:23.439819 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:23.442030 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:23.442499 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:23.443915 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:23.522853 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:23.527508 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:23.543808 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:23.560547 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:23.574056 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:23.578780 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:23.586719 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:23.619614 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:23.673991 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:23.762301 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:23.773856 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:23.775144 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:23.793924 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:23.820303 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:23.820308 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:23.874727 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:23.963613 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:23.990351 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:24.012688 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:24.023116 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:24.032999 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:24.042462 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:24.042912 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:24.050029 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:24.060570 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:24.112530 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(c130eb7ea8a74f4b8d9b79e23af70067): perf score=0.033354
W20240503 08:16:24.158056 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:24.158550 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:24.168740 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(c130eb7ea8a74f4b8d9b79e23af70067) complete. Timing: real 0.056s	user 0.041s	sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":5873,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":670,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":482,"threads_started":2,"wal-append.queue_time_us":341}
I20240503 08:16:24.169843 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(c130eb7ea8a74f4b8d9b79e23af70067): 4838 bytes on disk
I20240503 08:16:24.170403 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(c130eb7ea8a74f4b8d9b79e23af70067) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240503 08:16:24.183117 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:24.227684 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(da1f165b4a5647a48ad7ff951fb3fbf6): perf score=0.033373
I20240503 08:16:24.259362 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(da1f165b4a5647a48ad7ff951fb3fbf6) complete. Timing: real 0.031s	user 0.021s	sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1542,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":625,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":421,"threads_started":2,"wal-append.queue_time_us":306}
I20240503 08:16:24.260403 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(da1f165b4a5647a48ad7ff951fb3fbf6): 4583 bytes on disk
I20240503 08:16:24.260881 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(da1f165b4a5647a48ad7ff951fb3fbf6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:16:24.308879 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(f7a5221747794a13aa4ed763006ba291): perf score=0.033401
W20240503 08:16:24.339960 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:24.343159 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(f7a5221747794a13aa4ed763006ba291) complete. Timing: real 0.034s	user 0.023s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":369,"dirs.run_cpu_time_us":302,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":651,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1960,"threads_started":2,"wal-append.queue_time_us":1822}
I20240503 08:16:24.344244 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(f7a5221747794a13aa4ed763006ba291): 4923 bytes on disk
I20240503 08:16:24.344743 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(f7a5221747794a13aa4ed763006ba291) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240503 08:16:24.372534 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:24.441373 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:24.458784 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:24.474835 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:24.496695 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:24.504097 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:24.539558 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:24.545250 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:24.550133 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:24.570961 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:24.586125 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:24.610533 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:24.615600 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:24.634105 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:24.743618 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:24.749460 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:24.762267 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:24.764487 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:24.767812 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:24.772156 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:24.796348 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:24.851387 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240503 08:16:25.021502 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:25.024564 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:25.032574 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:25.041107 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:25.047981 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:25.053695 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:16:25.092502 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:25.112092 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:25.122660 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:25.133580 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:25.142680 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:25.163477 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:25.305473 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:16:25.306880 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(b5ceefc466f642d08ea96d549a534776): perf score=0.033336
W20240503 08:16:25.332020 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 08:16:25.354997 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(b5ceefc466f642d08ea96d549a534776) complete. Timing: real 0.048s	user 0.036s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":339,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1433,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":467,"threads_started":2,"wal-append.queue_time_us":347}
I20240503 08:16:25.356195 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(88404b9cdf574cb0bca1bb49348fb4cd): perf score=0.033345
I20240503 08:16:25.387171 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(88404b9cdf574cb0bca1bb49348fb4cd) complete. Timing: real 0.031s	user 0.021s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1659,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":527,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":245,"threads_started":1,"wal-append.queue_time_us":362}
I20240503 08:16:25.388411 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(b5ceefc466f642d08ea96d549a534776): 4923 bytes on disk
I20240503 08:16:25.388943 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(b5ceefc466f642d08ea96d549a534776) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:16:25.389984 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(88404b9cdf574cb0bca1bb49348fb4cd): 4583 bytes on disk
I20240503 08:16:25.390556 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(88404b9cdf574cb0bca1bb49348fb4cd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 08:16:25.425637 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(b5ceefc466f642d08ea96d549a534776): perf score=0.033368
I20240503 08:16:25.478013 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(b5ceefc466f642d08ea96d549a534776) complete. Timing: real 0.052s	user 0.026s	sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":315,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":2865,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":809,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14142,"threads_started":2,"wal-append.queue_time_us":14030}
I20240503 08:16:25.479471 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(b5ceefc466f642d08ea96d549a534776): 4923 bytes on disk
I20240503 08:16:25.480090 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(b5ceefc466f642d08ea96d549a534776) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 08:16:25.507941 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:25.517488 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(dfa711e8c8564bc78846b16d68470cea): perf score=0.033394
I20240503 08:16:25.544710 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:25.565865 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:25.570394 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:25.583197 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:25.627395 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:25.668041 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:25.677901 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:25.687717 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:25.735616 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(dfa711e8c8564bc78846b16d68470cea) complete. Timing: real 0.216s	user 0.060s	sys 0.035s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14859,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1547,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":954,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":45840,"threads_started":2,"wal-append.queue_time_us":31149}
I20240503 08:16:25.737082 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(dfa711e8c8564bc78846b16d68470cea): 4838 bytes on disk
I20240503 08:16:25.737689 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(dfa711e8c8564bc78846b16d68470cea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240503 08:16:25.816828 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:16:25.830508 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:25.879513 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:25.894774 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:25.894922 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:25.897588 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:25.905040 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:25.957806 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:25.966673 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:25.970106 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:25.981101 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:25.995738 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:26.030696 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:26.080061 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:26.174743 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:26.180454 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:26.188177 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:26.223973 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:26.259490 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:26.261031 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:26.281852 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:26.329035 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:26.469105 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:16:26.470008 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:26.475278 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:26.506891 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:26.519446 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:26.553061 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:26.569693 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:26.646483 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:26.647159 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:26.649170 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:26.667397 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:26.668313 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:26.702414 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:26.762845 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:26.772493 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:26.904583 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:26.933372 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:26.938885 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:26.964008 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:26.970356 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:26.970510 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:27.004851 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:27.010880 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:27.051220 14425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 96835.1us
W20240503 08:16:27.052150 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:27.055920 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:27.073228 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:27.105783 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:27.214020 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:27.245726 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:27.261900 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:27.264132 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:27.284822 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:27.292650 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:27.304287 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:27.313746 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:27.325377 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:27.457873 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:27.520388 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:27.571472 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:27.587010 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:27.696486 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:27.697611 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:27.701658 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:27.725879 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:27.744042 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:27.753535 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:27.886500 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:16:28.016021 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:28.033394 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:28.038620 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:28.069087 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:28.071696 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:28.077287 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:28.094378 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:28.102574 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:28.183241 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:28.267819 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:28.280048 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:28.281445 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:28.299444 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:28.302503 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:28.303417 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:28.303617 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:28.448957 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:28.463001 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:28.473708 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:28.517822 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:16:28.674239 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:28.698025 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:28.711759 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:28.714691 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:28.715564 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:28.717769 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:28.748564 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:28.758649 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:28.769692 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:28.838009 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:28.859062 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:28.957028 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:28.957887 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:28.972316 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:29.007252 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:29.020162 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:29.023283 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:29.034934 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:29.050454 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 98833.9us
W20240503 08:16:29.059206 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:29.105358 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:29.153422 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:29.230058 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:29.244800 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:29.250234 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:29.254630 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:29.258905 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:29.269353 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:29.270956 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:29.284250 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:29.301973 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:29.323678 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:29.402468 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:29.482106 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:29.553035 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:29.663239 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:29.667090 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:29.683859 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:29.697237 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:29.718822 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:29.751308 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:29.766501 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:29.766830 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:29.767380 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:29.792537 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:29.836293 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:29.927862 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 122 similar messages]
I20240503 08:16:29.933666 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:29.936509 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:29.958982 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:29.960837 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:29.972978 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:29.989284 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:30.003664 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:30.049049 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:30.054098 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 08:16:30.055017 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:30.177436 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:30.249329 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:30.258203 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:30.267277 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:30.287263 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:30.293768 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:30.307574 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:30.441429 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:30.445026 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:30.524013 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:30.560895 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:16:30.594928 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:30.689491 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:30.697703 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:30.700582 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:16:30.716123 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:30.731209 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:30.756206 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:30.766132 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:30.773797 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:30.845311 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:30.860201 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:16:30.962291 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:30.977540 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:30.978940 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:31.003993 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:31.031368 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:31.033653 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:31.057729 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:31.066737 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:31.071002 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:31.090418 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 47114.3us
W20240503 08:16:31.180387 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.235734 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:16:31.259469 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:31.263496 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:31.263729 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:31.274695 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:31.279044 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:31.282128 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:31.420903 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.479297 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:16:31.480295 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:31.486330 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:31.499866 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.578665 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:31.676883 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:31.682277 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:31.682983 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:31.707264 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:31.743438 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:31.774453 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.778614 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.792161 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:16:31.793573 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:31.811530 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.846462 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.851227 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:31.959621 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:31.998133 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:32.011107 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:32.022279 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:32.034210 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:32.037220 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:32.056558 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:32.056847 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:32.075322 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:32.128718 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:16:32.235577 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:32.242465 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:32.261041 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:32.270682 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:32.277360 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:16:32.291721 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:32.293161 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:32.366648 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:32.437161 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:32.472936 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:32.533162 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:32.563525 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:32.700668 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:32.710584 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:32.718596 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:32.736548 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:32.749121 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:32.751464 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:32.768702 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:32.860178 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:32.958189 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:32.981839 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:32.999985 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:33.009085 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:33.023331 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:33.038004 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:33.054414 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:33.078603 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:33.113229 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:33.164904 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:33.186394 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:16:33.190306 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:16:33.236439 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:33.236737 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:33.253597 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:33.255395 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:33.281798 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:33.288597 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:33.352705 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:33.483183 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:33.487308 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:33.507743 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:33.609935 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:33.675587 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:33.677561 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:33.677733 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:33.679271 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:33.682780 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:16:33.685259 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:33.705931 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:33.731539 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:33.860442 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:33.881043 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:33.892956 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:33.897895 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:33.897871 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:33.918825 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:33.954684 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:34.001048 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:34.093503 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:34.172191 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:34.193801 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:34.198006 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:34.217046 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:34.219465 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:34.237516 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:34.251389 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:16:34.267452 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:34.274585 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:34.298672 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:34.304451 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:34.306252 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:34.391891 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:34.557196 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:34.619004 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:34.657454 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:34.675727 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:34.678900 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:34.686738 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:34.687425 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:34.710018 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:34.738494 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:34.775152 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:34.815096 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:34.820761 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:16:34.913609 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:34.915120 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:34.921388 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:34.939922 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:34.950098 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240503 08:16:34.973445 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:34.980859 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:35.024995 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:35.095666 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:16:35.149715 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:16:35.157866 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:35.165557 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:16:35.185345 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:35.187163 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:35.190012 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:35.208070 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:35.209204 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:35.216913 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:35.229295 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:35.236904 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:35.556782 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:35.598374 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:16:35.625802 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:35.639114 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(f65ab6dba1124573962e2da58aa1859c): perf score=0.033383
I20240503 08:16:35.640432 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:35.649336 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:35.662452 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:35.680465 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:16:35.689435 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:35.695648 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:35.710675 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:35.717111 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:35.778682 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:35.779518 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:35.813771 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(f65ab6dba1124573962e2da58aa1859c) complete. Timing: real 0.174s	user 0.065s	sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":1646,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1220,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":887,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2098,"threads_started":3,"wal-append.queue_time_us":380}
I20240503 08:16:35.815961 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(f65ab6dba1124573962e2da58aa1859c): 4583 bytes on disk
I20240503 08:16:35.816799 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(f65ab6dba1124573962e2da58aa1859c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240503 08:16:35.823647 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:35.982038 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:36.022733 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:36.039234 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:36.068121 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:36.073300 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:36.090608 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:36.094081 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:36.103358 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:36.237874 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:36.268757 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:36.288111 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:36.302927 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:36.309481 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:36.345203 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:36.350251 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(897a701b880641b08816f2d1a254ef03): perf score=0.033355
I20240503 08:16:36.372063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:36.380362 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:36.413103 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:36.436589 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:36.465581 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:36.466035 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:36.502775 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240503 08:16:36.568728 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(897a701b880641b08816f2d1a254ef03) complete. Timing: real 0.218s	user 0.112s	sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":9039,"dirs.queue_time_us":8997,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":12331,"drs_written":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":67417,"threads_started":3,"wal-append.queue_time_us":49373}
I20240503 08:16:36.570982 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(897a701b880641b08816f2d1a254ef03): 4584 bytes on disk
I20240503 08:16:36.572794 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(897a701b880641b08816f2d1a254ef03) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20240503 08:16:36.609867 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:36.670874 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:36.690829 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:36.731189 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:36.785801 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:36.798612 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:36.824654 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(88404b9cdf574cb0bca1bb49348fb4cd): perf score=0.033377
I20240503 08:16:36.883457 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(88404b9cdf574cb0bca1bb49348fb4cd) complete. Timing: real 0.058s	user 0.039s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":2057,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8021,"threads_started":1,"wal-append.queue_time_us":7904}
I20240503 08:16:36.885620 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(88404b9cdf574cb0bca1bb49348fb4cd): 4583 bytes on disk
I20240503 08:16:36.886464 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(88404b9cdf574cb0bca1bb49348fb4cd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
W20240503 08:16:36.915011 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:36.956182 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:37.005048 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:37.055182 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:37.069968 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:37.099503 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:37.109475 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:37.135577 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:37.136541 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:37.169571 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:37.262562 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:37.366894 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:37.373988 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:37.379650 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:37.379998 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:37.395802 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:37.401535 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:37.406195 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:37.419922 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:37.424540 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:37.552397 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:37.624410 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:37.631381 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:37.634832 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:37.646358 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:37.647502 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:37.669250 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:37.683779 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:37.688066 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:37.783551 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:37.818456 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:38.006059 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:38.075809 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(b5ceefc466f642d08ea96d549a534776): perf score=0.033395
I20240503 08:16:38.099118 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:38.105593 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:38.110530 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:38.135324 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:38.141163 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:38.149756 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:38.175742 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:38.175908 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:16:38.177106 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:38.242655 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(b5ceefc466f642d08ea96d549a534776) complete. Timing: real 0.167s	user 0.042s	sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29361,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":2040,"drs_written":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4,"lbm_write_time_us":917,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42974,"threads_started":2,"wal-append.queue_time_us":6486}
I20240503 08:16:38.244052 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(a87c5af03f4d47d38bbfbd225c7195d1): perf score=0.033345
W20240503 08:16:38.268558 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:38.278815 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:16:38.284332 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(a87c5af03f4d47d38bbfbd225c7195d1) complete. Timing: real 0.040s	user 0.023s	sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1449,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":218,"threads_started":1,"wal-append.queue_time_us":332}
I20240503 08:16:38.285490 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(b5ceefc466f642d08ea96d549a534776): 4923 bytes on disk
I20240503 08:16:38.286063 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(b5ceefc466f642d08ea96d549a534776) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 08:16:38.287084 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(a87c5af03f4d47d38bbfbd225c7195d1): 4584 bytes on disk
I20240503 08:16:38.288174 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(a87c5af03f4d47d38bbfbd225c7195d1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 08:16:38.290594 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:38.315840 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:38.404083 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:38.419615 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:38.420379 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:38.435423 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:38.443691 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:38.445170 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:38.450337 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:38.469502 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:38.640595 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:38.648906 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(3fea213e5a12452f8b894b9cdb94f9ae): perf score=0.033396
W20240503 08:16:38.668864 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:38.687294 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:38.698050 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:38.702059 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:38.724418 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:38.731544 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:38.737893 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:38.750305 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:38.769773 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(3fea213e5a12452f8b894b9cdb94f9ae) complete. Timing: real 0.121s	user 0.041s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8083,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":15401,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16678,"threads_started":2,"wal-append.queue_time_us":8547}
I20240503 08:16:38.771337 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(c130eb7ea8a74f4b8d9b79e23af70067): perf score=0.033401
W20240503 08:16:38.795441 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:38.811523 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:38.829749 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(c130eb7ea8a74f4b8d9b79e23af70067) complete. Timing: real 0.056s	user 0.034s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":2365,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":229,"threads_started":1,"wal-append.queue_time_us":340}
I20240503 08:16:38.830987 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(3fea213e5a12452f8b894b9cdb94f9ae): 4923 bytes on disk
I20240503 08:16:38.831516 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(3fea213e5a12452f8b894b9cdb94f9ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:16:38.832523 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(c130eb7ea8a74f4b8d9b79e23af70067): 4838 bytes on disk
I20240503 08:16:38.833098 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(c130eb7ea8a74f4b8d9b79e23af70067) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 08:16:38.870684 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:38.928138 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:39.143172 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:39.146050 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:39.149233 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:39.170384 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:39.170781 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:39.179051 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:39.182327 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:39.189800 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:39.212554 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:39.311769 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:39.331686 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:39.352792 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:39.392508 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:39.403708 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:39.404356 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:39.414623 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:39.419723 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:39.429024 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:39.431968 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:39.457657 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:39.626808 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:39.640646 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:39.658130 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:39.662555 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:39.663555 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:39.678648 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:39.684341 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:39.698462 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:39.798816 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:39.798931 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:39.817724 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:39.939666 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:39.951988 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240503 08:16:40.075385 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:40.102524 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:40.120453 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:40.122918 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:40.127931 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:40.131281 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:40.134938 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:40.183203 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 08:16:40.270444 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:40.290798 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:40.330585 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:40.332688 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:40.342123 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:40.346783 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:40.355332 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:40.357966 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:40.371049 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:40.471496 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:40.545931 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(dd11460051ab40febc2fe6aff58b5028): perf score=0.033352
I20240503 08:16:40.579643 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:40.607772 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:40.624927 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:40.634349 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:40.638247 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:40.667320 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(dd11460051ab40febc2fe6aff58b5028) complete. Timing: real 0.120s	user 0.066s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":26573,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":1572,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1019,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":36741,"threads_started":2,"wal-append.queue_time_us":10138}
I20240503 08:16:40.668572 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(dd11460051ab40febc2fe6aff58b5028): 4838 bytes on disk
I20240503 08:16:40.669188 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(dd11460051ab40febc2fe6aff58b5028) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 08:16:40.696645 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:40.707556 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:40.715525 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:40.717070 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:40.738197 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:40.739140 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:40.819027 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:40.847802 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:40.881669 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:16:40.973697 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:40.983289 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:40.998859 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:16:41.107766 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:41.108639 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:41.114470 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:41.116343 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:41.128270 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:41.138573 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:41.142495 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:41.171327 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:41.212836 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:41.335821 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:41.343330 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(b19df3ae9f2b44a3818855f259962fff): perf score=0.033402
I20240503 08:16:41.359328 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:41.360965 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:41.364070 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:41.376261 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:41.421849 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(b19df3ae9f2b44a3818855f259962fff) complete. Timing: real 0.078s	user 0.048s	sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":437,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":9288,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1222,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8179,"threads_started":2,"wal-append.queue_time_us":7664}
I20240503 08:16:41.423514 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(6b677894b51f4dc58f157d8f4d632e9a): perf score=0.033364
I20240503 08:16:41.424995 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:41.427259 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:41.471001 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:41.493803 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(6b677894b51f4dc58f157d8f4d632e9a) complete. Timing: real 0.070s	user 0.038s	sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":805,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21205,"threads_started":1,"wal-append.queue_time_us":21105}
I20240503 08:16:41.495138 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(6b677894b51f4dc58f157d8f4d632e9a): 4923 bytes on disk
I20240503 08:16:41.495817 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(6b677894b51f4dc58f157d8f4d632e9a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 08:16:41.496876 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(b19df3ae9f2b44a3818855f259962fff): 4583 bytes on disk
I20240503 08:16:41.497421 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(b19df3ae9f2b44a3818855f259962fff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 08:16:41.543146 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:41.618057 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:41.640352 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:41.648453 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:41.649166 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:16:41.653719 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:41.654805 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:41.663774 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:41.673379 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(562e1ec5c34945298cf457758328057f): perf score=0.033396
W20240503 08:16:41.677744 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:41.689080 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:41.692158 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:41.715262 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:41.756578 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:41.756794 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:41.760406 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(562e1ec5c34945298cf457758328057f) complete. Timing: real 0.087s	user 0.033s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":397,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":14412,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13754,"threads_started":2,"wal-append.queue_time_us":7410}
I20240503 08:16:41.761559 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(562e1ec5c34945298cf457758328057f): 4583 bytes on disk
I20240503 08:16:41.762351 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(562e1ec5c34945298cf457758328057f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 08:16:41.894415 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:41.930723 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:42.014319 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(12df78f752c34fe3885b235c4a19e325): perf score=0.033367
I20240503 08:16:42.071857 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(12df78f752c34fe3885b235c4a19e325) complete. Timing: real 0.057s	user 0.043s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1762,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":884,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":309,"threads_started":1,"wal-append.queue_time_us":480}
I20240503 08:16:42.072983 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(12df78f752c34fe3885b235c4a19e325): 4583 bytes on disk
I20240503 08:16:42.073643 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(12df78f752c34fe3885b235c4a19e325) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240503 08:16:42.129621 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:42.134562 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:42.155416 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:42.156424 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:42.160413 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:42.167532 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:42.171211 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:42.190905 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:42.195626 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:42.196838 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:42.315517 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:42.325598 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(1d6c920958214148b3c3d5ac995e99e2): perf score=0.033351
I20240503 08:16:42.376117 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(1d6c920958214148b3c3d5ac995e99e2) complete. Timing: real 0.050s	user 0.040s	sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1568,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":295,"threads_started":1,"wal-append.queue_time_us":419}
I20240503 08:16:42.377049 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(1d6c920958214148b3c3d5ac995e99e2): 4923 bytes on disk
I20240503 08:16:42.377662 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(1d6c920958214148b3c3d5ac995e99e2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 08:16:42.395395 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:42.398253 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:42.402227 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:42.421586 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:42.423028 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:42.423311 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:42.623042 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:42.631583 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(7aa1031b9e30478597da5ad3708bdff5): perf score=0.033362
I20240503 08:16:42.636044 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:42.637770 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:42.658116 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:42.672399 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:42.683036 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:42.684557 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:42.719297 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:42.720105 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:42.737309 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:42.753268 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(1d6c920958214148b3c3d5ac995e99e2): perf score=0.033385
I20240503 08:16:42.769754 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(7aa1031b9e30478597da5ad3708bdff5) complete. Timing: real 0.138s	user 0.073s	sys 0.044s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1283,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1545,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1269,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":75612,"threads_started":1,"wal-append.queue_time_us":76013}
I20240503 08:16:42.771790 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(7aa1031b9e30478597da5ad3708bdff5): 4923 bytes on disk
I20240503 08:16:42.787842 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(7aa1031b9e30478597da5ad3708bdff5) complete. Timing: real 0.006s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":6052,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 08:16:42.827093 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(1d6c920958214148b3c3d5ac995e99e2) complete. Timing: real 0.074s	user 0.029s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6079,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":1750,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":929,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6191,"threads_started":2,"wal-append.queue_time_us":320}
I20240503 08:16:42.828187 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(1d6c920958214148b3c3d5ac995e99e2): 4923 bytes on disk
I20240503 08:16:42.828790 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(1d6c920958214148b3c3d5ac995e99e2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 08:16:43.032261 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:43.119944 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:43.132596 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:43.151048 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:43.168538 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:43.168978 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:43.175865 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:43.176328 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:43.178763 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:16:43.197887 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:43.199739 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:43.215544 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:43.269407 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:43.319664 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:16:43.369848 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:43.388546 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:43.398838 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:43.420863 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:43.426820 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:43.434724 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:43.442318 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:43.463996 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:43.496212 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:43.562433 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:43.644202 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:43.645974 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:43.654559 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:43.671775 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:43.678900 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:43.712683 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:43.725759 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:43.734405 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:43.763532 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:43.790858 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:43.812095 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:43.982261 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:44.014869 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.105307 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:44.108238 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:44.108798 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:44.109802 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:44.137187 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:44.152330 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:44.153167 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:44.231420 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:44.249554 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.293905 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(5d16c2981213464cbefa44b6ca509182): perf score=0.033343
I20240503 08:16:44.335479 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(0780fb979eee4bae98a014d9721c9441): perf score=0.033401
I20240503 08:16:44.350812 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(5d16c2981213464cbefa44b6ca509182) complete. Timing: real 0.057s	user 0.053s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6876,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":4637,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7281,"threads_started":2,"wal-append.queue_time_us":337}
I20240503 08:16:44.352386 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(5d16c2981213464cbefa44b6ca509182): 4924 bytes on disk
I20240503 08:16:44.354635 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(5d16c2981213464cbefa44b6ca509182) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 08:16:44.374699 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.381268 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:44.386003 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:44.393625 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:44.418866 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:44.423696 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:44.434937 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.459450 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:44.462395 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:44.471679 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.515849 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(0780fb979eee4bae98a014d9721c9441) complete. Timing: real 0.180s	user 0.061s	sys 0.045s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":26625,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26783,"threads_started":2,"wal-append.queue_time_us":367}
I20240503 08:16:44.517225 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(0780fb979eee4bae98a014d9721c9441): 4838 bytes on disk
I20240503 08:16:44.517868 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(0780fb979eee4bae98a014d9721c9441) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 08:16:44.631139 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:44.659916 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.684873 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:44.709976 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:44.717912 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:44.718650 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:44.742177 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:44.750471 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:44.750609 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:44.766463 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:44.769788 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(f19b7ef418974b06bdcc7125e8cf732d): perf score=0.033364
I20240503 08:16:44.830008 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(f19b7ef418974b06bdcc7125e8cf732d) complete. Timing: real 0.060s	user 0.035s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":2308,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":279,"threads_started":1,"wal-append.queue_time_us":417}
I20240503 08:16:44.831307 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(f19b7ef418974b06bdcc7125e8cf732d): 4838 bytes on disk
I20240503 08:16:44.832083 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(f19b7ef418974b06bdcc7125e8cf732d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 08:16:44.947471 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:44.960482 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:45.094358 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:16:45.146730 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:45.174032 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:45.178102 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:45.186429 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:45.196611 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:16:45.207417 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:45.209491 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:45.211771 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:45.235306 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:45.247993 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:45.408576 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:45.412477 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:45.432153 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:45.432379 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:45.447000 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:45.465667 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:45.503029 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:45.523422 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:45.638864 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:45.658954 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:45.658991 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:45.667234 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:45.679868 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:45.687566 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:45.699877 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:45.748533 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:45.759567 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:45.765798 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:16:45.767670 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:45.888896 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:45.917392 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:45.929280 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:45.999332 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:46.011797 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:46.088042 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:46.108073 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:46.112354 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:16:46.119841 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:46.120241 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:46.123064 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:46.127717 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:46.155879 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:46.181418 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:46.295801 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:46.333065 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:46.344326 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:46.348368 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:46.354636 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:46.357082 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:46.373862 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:46.391177 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:46.425300 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:46.436686 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:46.556739 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:46.598749 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:46.600885 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:46.612042 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:46.621250 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:46.627787 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:46.645790 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:46.646035 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:46.692026 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:16:46.768908 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:46.816045 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:46.852283 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:47.020915 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:47.050483 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:47.090452 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:47.100574 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:47.109025 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:47.116009 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:47.122516 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:47.136413 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:47.151823 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:47.174554 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:47.201735 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:47.226485 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:47.278785 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:47.348578 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:47.363955 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:47.365437 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:47.367197 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:47.387317 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:47.390492 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:47.601500 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:47.616329 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:47.633158 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:47.633608 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:47.661001 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:47.676352 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:47.691216 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:47.698513 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:47.710485 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:47.718487 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:47.774717 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:47.952807 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:16:48.081611 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:48.087776 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:48.092612 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:48.094398 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:48.100260 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:48.117030 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:48.125604 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:48.126091 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:48.133005 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:48.186300 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:48.213945 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:48.230381 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:48.282667 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:48.349483 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:48.366672 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:48.367314 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:48.374444 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:48.383514 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:48.384944 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:48.405300 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:48.539350 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:48.561373 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:48.618304 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:48.628243 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:16:48.632165 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:48.638468 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:16:48.646471 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:48.650396 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:48.675765 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:48.682967 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:48.692612 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:48.737448 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:48.898192 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:48.928324 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:48.984385 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:48.994437 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:49.082336 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:49.113457 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:49.119145 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:49.124506 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:49.130537 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:49.130915 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:16:49.134272 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:49.146641 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:49.147029 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:49.350651 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:49.359525 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:49.371225 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:49.385896 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:49.405455 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:49.415343 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:49.418078 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:49.474109 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:49.525367 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:49.530464 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:49.611979 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:16:49.615603 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:49.625435 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:49.634322 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:49.650017 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:49.651062 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:49.660187 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:49.749527 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:49.832267 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:49.909833 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:49.911142 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:49.952863 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:50.044875 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:50.066309 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:50.076207 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:50.083216 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:50.089305 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:50.100550 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 118 similar messages]
I20240503 08:16:50.108211 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:50.113539 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:50.125387 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:50.194466 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:50.207924 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:50.289860 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 08:16:50.314371 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:50.314985 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:50.326457 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:50.336627 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:50.355901 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:50.365454 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:50.397945 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:50.557482 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:50.564705 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:50.579759 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:50.595671 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:50.595839 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:50.614516 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:50.625478 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:50.625922 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:50.651769 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:50.662510 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:50.665066 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:16:50.674981 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:50.688918 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:50.816398 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:50.923801 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:16:50.942075 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:51.029846 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:16:51.082551 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:51.088908 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:51.100069 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:16:51.112125 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:51.112905 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:51.117780 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:51.118264 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:51.198935 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:51.316059 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:16:51.339569 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:51.351758 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:51.352171 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:51.354197 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:51.354539 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:51.383023 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:51.400831 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:51.430428 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:51.437063 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:51.472451 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:51.574471 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:16:51.603022 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:51.604846 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:51.606549 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:16:51.625448 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:51.627048 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:51.633134 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:51.644325 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:51.779955 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:51.821722 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:16:51.822635 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:51.886268 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:52.060081 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:52.081429 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:52.092832 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:52.093739 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:52.119102 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:52.123044 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:52.202490 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:52.299830 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:52.313118 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:52.326802 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:52.332535 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:52.344606 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:52.348554 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:52.358968 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:52.367712 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:52.368552 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:52.377086 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:52.431785 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:52.483321 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:52.500568 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:52.548188 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:52.551601 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:52.565074 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:52.572938 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:52.595018 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:16:52.645257 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:52.647423 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:52.692133 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:52.710383 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:52.916641 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:53.000926 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:53.033949 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:53.070206 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:16:53.078809 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:53.082249 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:53.091552 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:53.104813 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:53.112740 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:53.119488 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:53.159511 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:53.160367 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:53.202620 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:53.300523 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:53.340191 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:53.351624 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:53.371261 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:53.371274 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:53.386562 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:53.400712 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:16:53.411121 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:53.413872 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:53.491134 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:53.508096 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:53.617128 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:53.632294 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:53.633996 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:53.638455 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:53.650494 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:53.651994 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:53.678489 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:53.704335 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:53.782501 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:16:53.872159 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:53.924974 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:53.943861 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:54.044831 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:54.048049 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:54.070299 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:54.094282 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:54.105314 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:54.111109 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:54.120102 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:54.125927 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:54.160585 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:54.187739 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:54.342813 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:54.357939 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:54.374322 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:54.381301 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:54.402629 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:54.406128 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:54.427840 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:54.600991 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:54.605239 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:54.614496 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:54.628484 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:54.636834 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:54.651274 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:54.661600 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:54.667362 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:54.677189 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:54.704277 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:54.773262 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:54.873721 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:54.974270 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:55.077015 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:55.086251 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:55.088660 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:55.099095 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:55.116858 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:16:55.118176 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:55.123235 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:55.123775 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:55.178278 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:55.179035 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:55.226279 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:55.239590 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:55.300069 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:55.307785 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:55.312181 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:55.312762 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240503 08:16:55.320215 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:55.332439 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:55.359679 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:55.360848 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:55.441890 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:55.563867 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:55.576426 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:55.581221 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:55.593809 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:55.595809 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:16:55.648725 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:55.651530 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:16:55.655586 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:55.718384 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:55.755921 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:16:55.769455 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:55.846720 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:55.862046 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:55.935566 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:55.946198 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:16:56.018254 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:56.051796 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:56.055729 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:56.066699 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:56.075826 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:56.080902 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:56.086946 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:56.182889 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:56.211875 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:56.273838 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:16:56.300365 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:56.303546 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:56.306921 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:56.329970 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:56.354795 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:56.362824 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:56.379490 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:56.391649 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:56.442214 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:16:56.555766 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:56.567999 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:56.574046 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:56.586839 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:56.616873 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:56.619197 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:16:56.622136 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:56.669309 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:56.708779 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:56.727973 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:56.866410 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:16:56.886039 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:57.005193 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:57.027077 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:57.027180 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:57.038918 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:57.054349 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:57.054416 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:57.076372 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:57.103691 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:57.122485 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:57.202525 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:57.215531 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:16:57.258105 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:57.269083 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:57.272293 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:57.275611 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:57.283102 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:57.283958 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:16:57.307812 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:57.312866 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:57.506431 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:57.540877 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:57.558967 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:16:57.574323 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:57.575575 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:57.598189 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:57.646091 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:57.646091 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:57.820225 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:57.823941 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:57.849704 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:57.875871 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:57.876304 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:58.001237 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:58.053756 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:58.054901 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:58.059291 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:58.063855 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:58.068007 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:58.077379 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:58.077695 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:58.097218 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:58.141314 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:58.149582 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:16:58.302960 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:58.307787 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:58.312428 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:58.315933 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:58.324709 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:58.328493 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:58.375701 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:58.409415 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:58.468909 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:58.476039 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:58.517541 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:16:58.532018 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:58.533031 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:58.542713 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:58.560990 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:58.566780 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:58.581985 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:58.610517 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:58.799671 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:16:58.893812 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:58.905207 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:58.956007 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:58.977865 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:59.000255 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:59.028633 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:59.028926 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:59.033452 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:59.040729 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:59.055794 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:59.077713 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:59.224552 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:59.246892 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:59.268045 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:59.268842 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:59.271798 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:16:59.272387 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:16:59.295307 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:16:59.314778 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:16:59.367067 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:59.412606 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:59.495427 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:16:59.514447 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:59.514456 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:16:59.522553 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:16:59.536357 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:59.547780 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:16:59.551381 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:59.557669 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:16:59.674767 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:59.714378 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:59.741493 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:16:59.824466 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:16:59.872674 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:16:59.964224 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:16:59.973181 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:16:59.974620 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:00.003155 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:00.012553 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:00.047544 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:00.049038 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:00.084261 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:17:00.100553 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:17:00.129720 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:17:00.176998 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:00.236933 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:00.245756 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:00.253526 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:00.266769 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:00.286744 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:00.286880 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:00.292575 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:00.454313 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:00.455601 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:17:00.456925 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:17:00.475349 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:00.479979 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:00.489290 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:00.509909 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:00.527660 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:00.543774 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:00.551054 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:17:00.584765 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:00.673450 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:00.739784 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:00.783999 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:00.826220 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:17:00.868580 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:17:00.897035 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:00.922201 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:00.931902 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:00.935703 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:00.938019 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:00.953372 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:01.017840 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:01.018364 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:01.036538 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:01.067087 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:17:01.185499 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:01.189950 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:01.191860 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:01.200014 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:01.201332 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:01.202631 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:01.208086 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:01.222995 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:01.272292 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:01.348893 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:17:01.384871 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:17:01.419032 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:01.447670 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:01.448349 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:01.462855 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:01.484066 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:01.490209 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:01.498853 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:01.533478 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:01.603819 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:01.643702 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:01.771234 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:01.900741 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:01.913048 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:01.930974 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:01.961752 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:01.966993 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:01.979420 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:02.022295 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:02.036511 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:02.044158 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:02.065995 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:02.092200 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:17:02.109167 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:02.161067 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:02.180356 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:02.210716 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:02.216207 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:02.217675 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:02.231108 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:02.250696 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:02.266610 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:02.340839 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:02.444252 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:02.475452 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:02.481935 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:02.486501 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:02.499084 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:17:02.530404 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:02.530511 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:02.538960 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:02.684300 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:02.728049 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:02.796875 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:02.902079 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:02.977124 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:02.989683 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:03.005595 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:17:03.006047 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:03.007354 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:03.033303 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:03.044695 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:03.077448 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:03.107344 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:03.172086 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:03.235500 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:03.245282 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:03.273825 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:03.290789 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:17:03.307824 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:03.310236 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:03.326880 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:17:03.335418 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:03.355527 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:03.359001 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:03.389112 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:03.411226 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:03.472966 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:03.519956 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:03.530593 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:03.536024 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:03.549708 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:03.576524 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:03.589623 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:03.714499 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:03.788111 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:17:03.827422 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:03.865867 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:03.905102 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:17:03.979135 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:04.000489 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:04.003929 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:04.014554 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:17:04.025120 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:04.047698 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:04.053208 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:04.065958 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:04.067798 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:04.232085 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:04.235847 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:04.255095 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:04.255651 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:04.259227 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:04.282500 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:04.283187 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:04.296245 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:04.422438 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:04.444381 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:04.480917 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:04.482287 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:04.495112 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:04.520367 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:04.550191 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:04.554008 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:04.568356 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:04.710726 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:04.722573 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:04.728926 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:04.851662 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:04.861505 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:04.913702 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:04.949959 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:17:04.967891 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:04.982861 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:04.997325 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:05.027436 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:05.043401 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:05.069347 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:05.168687 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240503 08:17:05.174948 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:05.253494 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:05.261869 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:05.268784 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:05.269430 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:05.294008 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:05.302261 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:05.309916 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:05.346634 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:05.395956 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:05.465137 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:17:05.517617 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:05.539355 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:05.539476 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:05.549291 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:05.598100 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:17:05.600456 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:05.620589 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:05.622174 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:05.685120 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:05.685356 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:05.699080 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:05.730988 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:05.760008 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:05.781399 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:05.891837 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:05.973591 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:05.999259 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:06.007220 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:17:06.007654 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:06.032202 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:06.050040 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:06.050338 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:06.054966 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:06.055989 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:06.095816 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:06.109635 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:06.248226 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:06.259244 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:06.262637 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:06.280459 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:06.299604 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:17:06.300122 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:06.300961 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:06.403090 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:06.455744 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:06.486121 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:06.498229 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:06.500140 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:06.519809 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:06.530436 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:06.546563 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:06.549098 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:06.577129 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:06.610426 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:06.716738 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:06.860294 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:06.959647 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:06.991542 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:07.020349 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:07.024844 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:07.039189 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:07.050593 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:07.055419 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:17:07.069304 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:07.074429 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:07.081126 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:07.086513 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:07.131217 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:07.199062 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:17:07.301216 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:07.302884 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:07.311178 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:07.320200 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:07.338023 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:07.338224 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:07.352367 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:07.424904 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:07.480696 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:07.530483 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:07.553113 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:07.555435 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:07.564888 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:07.576486 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:07.621662 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:07.625401 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:07.649878 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:07.680481 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:07.749918 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:07.802223 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:08.009383 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:08.022040 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:08.035379 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:08.047732 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:08.063026 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:08.075964 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:08.090226 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:08.090381 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:08.107311 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:08.115501 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:17:08.120368 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:08.122779 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:08.127701 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:08.227118 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:08.274914 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:17:08.287184 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:08.293192 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:08.301730 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:08.311432 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:17:08.315858 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:08.328414 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:08.329275 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:08.338449 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:08.548895 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:08.565184 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:08.578263 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:08.578413 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:08.597337 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:08.608773 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:08.615849 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:08.625684 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:08.674726 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:08.678968 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:08.739996 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:17:08.987476 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:09.029094 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:09.041162 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:09.045758 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:09.068120 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:09.102125 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:09.122473 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:09.156644 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:09.180330 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:09.182651 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:09.243268 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:09.277894 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:09.310518 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:09.327011 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:09.328965 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:09.352121 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:09.354087 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:09.371044 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:09.373652 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:09.540668 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:09.561019 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:09.583531 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:09.588981 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:09.589916 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:09.606961 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:09.631229 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:09.649777 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:09.661820 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:09.677862 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:09.741376 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:09.769517 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:10.037235 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:10.052160 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:10.062947 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:10.085228 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:10.116164 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:10.117389 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:10.140383 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:10.171010 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:17:10.171979 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:10.172232 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:10.248688 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:10.271037 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:10.293757 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:10.308465 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:10.310883 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:10.314795 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:10.362761 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:10.364884 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:10.367705 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:10.377990 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:10.450359 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:10.539497 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:17:10.562714 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:10.564574 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:10.571666 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:10.572077 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:10.600351 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:17:10.602072 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:10.653920 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:10.674706 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:10.688617 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:10.692358 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:10.736047 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:10.764904 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:10.767374 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:10.795755 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:11.042692 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:17:11.083062 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:11.092662 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:11.097298 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 08:17:11.103495 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:11.115816 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:11.133512 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:11.134310 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:11.140936 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:11.146241 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:11.177342 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:11.177877 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:17:11.236291 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:11.331780 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:11.342540 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:11.354189 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:11.367651 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:11.369684 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:11.384865 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:11.494210 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:11.514636 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:11.538141 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:11.540091 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:17:11.616777 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:11.624683 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:11.628399 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:11.639770 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:11.661191 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:11.668529 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:11.683187 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:11.685957 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(5c595bb19d6946b0b1f853064892e946): perf score=0.033375
W20240503 08:17:11.705914 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:11.755406 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(5c595bb19d6946b0b1f853064892e946) complete. Timing: real 0.069s	user 0.042s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6886,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":933,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":802,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7035,"threads_started":2,"wal-append.queue_time_us":355}
I20240503 08:17:11.756865 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(5c595bb19d6946b0b1f853064892e946): 4838 bytes on disk
I20240503 08:17:11.757489 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(5c595bb19d6946b0b1f853064892e946) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
W20240503 08:17:11.793704 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:11.924654 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:11.929333 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:12.076750 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:12.078152 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:12.090996 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:12.096184 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:12.101620 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:12.112202 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:12.137712 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:12.168992 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:12.329169 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:12.329473 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:12.340165 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:12.343781 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:12.361595 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:17:12.383998 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:12.390762 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:12.407621 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:12.422550 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:12.448696 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:12.569675 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:12.577244 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:17:12.634537 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:12.656188 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:12.672681 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:12.673975 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:12.674307 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:12.677528 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:12.691380 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:12.704450 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:12.817127 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:12.845966 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:12.857532 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:13.028486 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:13.042604 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:13.046125 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:17:13.094254 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:13.096014 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:13.100888 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:13.112478 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:13.122674 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:13.169186 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:13.170621 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:13.216722 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:13.293503 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:13.363668 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:13.370263 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:13.375892 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:13.382009 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:13.401202 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:13.416939 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:13.420490 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:13.434188 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:13.503764 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:13.519093 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:13.574927 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:13.632669 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:13.644678 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:13.645469 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:13.645933 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:13.652096 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:13.676061 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:13.691833 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:17:13.734289 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:13.844269 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:13.881568 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:13.956354 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:13.999850 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:14.072039 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:14.093632 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:14.126009 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:14.139086 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:14.153227 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:14.161757 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:14.169169 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:14.173743 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:14.194721 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:14.307718 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:17:14.367259 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:14.369824 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:14.377431 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:14.384811 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:14.403091 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:14.419936 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:14.463526 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:14.474318 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:14.480723 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:14.635555 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:14.636529 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:14.637404 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:14.646027 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:14.664578 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:14.668846 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:17:14.669569 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:14.918840 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:14.948415 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:14.972110 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:14.982756 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:14.994828 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:15.089799 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:15.093715 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:15.102919 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:15.111878 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:17:15.122936 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:15.133869 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:15.201726 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:15.310461 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:15.352196 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:15.369621 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:17:15.370766 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:17:15.388207 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:15.388468 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:15.389233 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:15.391028 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:15.402630 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:17:15.416771 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:15.425910 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:15.460565 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:15.549314 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:17:15.569320 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:15.571204 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:15.593585 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:15.596284 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:15.619747 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:15.622527 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:15.638103 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:15.638790 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:15.644604 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:15.671826 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:15.967084 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:15.983399 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:17:16.036033 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:16.122809 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:16.123478 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:16.126597 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:16.133042 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:16.137390 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:16.139695 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:16.146472 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:17:16.158116 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:16.183540 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:16.194589 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:16.222684 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:16.263298 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:16.305351 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:16.374279 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:16.415287 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:16.425426 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:16.434044 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:16.440331 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:16.441630 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:16.463405 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:16.486130 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:16.529124 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:16.571250 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:16.609764 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:16.665580 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:16.673036 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:16.684177 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:16.685030 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:16.685403 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:16.734262 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:16.750092 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:16.967617 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:17.037636 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:17.055356 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:17.120462 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:17.129709 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:17.138399 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:17.172418 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:17.173265 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:17.174470 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:17.188397 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:17.229410 14817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.2.49.134:36937, user_credentials={real_user=slave}} blocked reactor thread for 47575us
W20240503 08:17:17.230973 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:17.377223 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:17.382514 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:17.389261 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:17.399760 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:17.400174 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:17.462344 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:17.472434 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:17.569352 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:17.570356 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:17.612190 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:17.623229 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:17.642660 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:17.650861 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:17.661846 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:17.670275 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:17.671662 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:17.671936 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:17.674932 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:17.706987 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:17.708479 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:17.726765 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling FlushMRSOp(f7a5221747794a13aa4ed763006ba291): perf score=0.033343
I20240503 08:17:17.776273 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: FlushMRSOp(f7a5221747794a13aa4ed763006ba291) complete. Timing: real 0.049s	user 0.030s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4429,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1060,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":768,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4577,"threads_started":2,"wal-append.queue_time_us":359}
I20240503 08:17:17.777510 16451 maintenance_manager.cc:419] P 0f8702fad8ea4d22840dab11bb27f7f2: Scheduling UndoDeltaBlockGCOp(f7a5221747794a13aa4ed763006ba291): 4923 bytes on disk
I20240503 08:17:17.778067 16342 maintenance_manager.cc:643] P 0f8702fad8ea4d22840dab11bb27f7f2: UndoDeltaBlockGCOp(f7a5221747794a13aa4ed763006ba291) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 08:17:17.861347 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:17.958837 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:17.971925 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:17.978837 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:18.010824 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:18.133620 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:18.153329 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:18.154058 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:18.160212 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:18.179311 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:18.180590 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:18.185413 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:18.201157 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:18.232867 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:18.378892 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:18.410683 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:18.422227 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:18.432672 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:18.433281 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:18.437415 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:18.443109 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:18.450690 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:17:18.475443 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(897a701b880641b08816f2d1a254ef03): perf score=0.033380
W20240503 08:17:18.510274 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:17:18.559384 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:18.567268 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(897a701b880641b08816f2d1a254ef03) complete. Timing: real 0.091s	user 0.042s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1026,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":900,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6620,"threads_started":2,"wal-append.queue_time_us":6283}
I20240503 08:17:18.568919 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(897a701b880641b08816f2d1a254ef03): 4584 bytes on disk
I20240503 08:17:18.569463 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(897a701b880641b08816f2d1a254ef03) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240503 08:17:18.649215 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:18.659962 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:18.681368 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:18.699409 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:18.706475 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:18.734040 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:18.734717 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:18.776244 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:18.791851 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:18.797716 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:18.818890 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:18.859582 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:18.860554 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:18.936630 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:19.108968 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:19.159740 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:19.177259 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:19.178726 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:19.191430 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:19.196195 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:19.221158 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(30b13cdf4b234f17872f5d1588915968): perf score=0.033400
I20240503 08:17:19.229781 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:19.241680 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:19.311533 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(30b13cdf4b234f17872f5d1588915968) complete. Timing: real 0.090s	user 0.051s	sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":418,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":992,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":743,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":496,"threads_started":2,"wal-append.queue_time_us":340}
I20240503 08:17:19.312897 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(169bd9ea6772420abf757dd55cf44d31): perf score=0.033375
W20240503 08:17:19.313153 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:19.347147 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(169bd9ea6772420abf757dd55cf44d31) complete. Timing: real 0.034s	user 0.027s	sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":261,"threads_started":1,"wal-append.queue_time_us":372}
I20240503 08:17:19.348857 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(169bd9ea6772420abf757dd55cf44d31): 4923 bytes on disk
I20240503 08:17:19.349490 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(169bd9ea6772420abf757dd55cf44d31) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:17:19.350847 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(30b13cdf4b234f17872f5d1588915968): 4838 bytes on disk
I20240503 08:17:19.351490 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(30b13cdf4b234f17872f5d1588915968) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
W20240503 08:17:19.399205 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:19.422061 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:19.431252 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:19.441629 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:19.446719 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:19.480212 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:19.482806 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:19.513535 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:19.628736 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(88404b9cdf574cb0bca1bb49348fb4cd): perf score=0.033354
I20240503 08:17:19.672780 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(88404b9cdf574cb0bca1bb49348fb4cd) complete. Timing: real 0.044s	user 0.023s	sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":515,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1882,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":759,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4685,"threads_started":2,"wal-append.queue_time_us":4251}
I20240503 08:17:19.675508 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(88404b9cdf574cb0bca1bb49348fb4cd): 4583 bytes on disk
I20240503 08:17:19.676201 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(88404b9cdf574cb0bca1bb49348fb4cd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 08:17:19.686806 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:19.705854 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:19.717128 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:19.721958 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:19.746246 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:19.765273 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:19.768391 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:19.868119 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:19.917099 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:19.973506 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:19.986608 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:20.008404 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:20.030639 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:20.170447 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:20.172807 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:20.186349 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:20.199879 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:20.214133 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:20.231783 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:20.246143 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:20.261093 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:20.346916 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:20.363842 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:20.407904 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240503 08:17:20.440106 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:20.446348 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:20.448040 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:20.451786 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:20.467171 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:20.471112 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:20.474493 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:20.476159 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:20.518673 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:20.525763 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:20.530947 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:20.647217 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:20.667061 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:20.692668 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:20.702517 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:20.711727 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:20.741012 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:20.792780 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:20.792779 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:20.859124 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:17:20.879871 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:20.934947 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:21.047454 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:21.050951 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:21.061679 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:21.126523 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:21.160558 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:21.173851 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:21.181519 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:21.182135 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:17:21.198539 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:21.202940 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:21.222334 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 08:17:21.231925 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:21.282279 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:21.319533 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:21.359835 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(0780fb979eee4bae98a014d9721c9441): perf score=0.033399
W20240503 08:17:21.363590 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:21.397640 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:21.405928 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(0780fb979eee4bae98a014d9721c9441) complete. Timing: real 0.046s	user 0.033s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":631,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1845,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1418,"threads_started":2,"wal-append.queue_time_us":1290}
I20240503 08:17:21.408296 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(0780fb979eee4bae98a014d9721c9441): 4838 bytes on disk
I20240503 08:17:21.409030 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(0780fb979eee4bae98a014d9721c9441) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240503 08:17:21.414688 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:21.425375 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:21.436499 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:21.436920 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:21.463675 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:21.469311 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:21.491633 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:21.640909 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:21.686338 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:21.702587 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:21.704939 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:21.725091 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:21.729025 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:21.755280 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:21.805245 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:21.900300 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:21.933377 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:22.129211 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:22.146493 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:22.154570 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:22.159600 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:22.160917 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:22.182850 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:22.184212 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:22.195281 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:22.218935 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:22.267436 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:22.363936 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:22.385560 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:22.409464 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:22.410176 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:22.430544 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:22.433012 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:22.449138 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:22.484267 14816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 64389us
I20240503 08:17:22.501955 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:22.505406 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:22.535408 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:22.558782 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:17:22.716775 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:22.727147 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:22.734489 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:22.753114 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:22.761209 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:22.791339 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:22.803467 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:22.803463 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:22.804476 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:22.814338 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:22.815608 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:22.895466 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:22.954867 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:22.958457 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:23.125087 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:23.165128 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:23.192747 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:23.205036 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:23.205261 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:23.206259 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:23.208050 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:23.229923 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:23.265383 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:23.330672 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:23.432353 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:17:23.438601 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:23.444666 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:23.475406 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:23.483040 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:23.504038 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:23.514307 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:23.550465 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:23.559772 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:23.631784 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:23.643605 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:23.677525 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:23.758858 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:23.775936 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:23.783821 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:23.784916 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:17:23.787695 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:23.788412 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:23.791312 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:17:23.793826 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:23.799088 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:23.941140 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:24.182672 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:24.188943 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:24.196565 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:24.222801 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:24.224174 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:24.233070 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:24.236253 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:24.251138 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:24.251307 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:24.277345 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:24.424165 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:17:24.466614 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:24.486922 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:17:24.489913 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:24.492556 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:24.510613 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:24.578204 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:24.613571 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:24.615110 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:24.669354 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:24.788790 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:24.794104 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:24.803754 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:24.803886 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:24.805166 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:24.824832 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:24.826645 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:24.843200 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:24.898764 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:24.910442 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.079036 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.226403 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:25.262466 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:25.279719 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:25.282055 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:25.291884 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:25.316823 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:25.317929 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:25.318552 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:25.318348 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.321063 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.322974 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.339048 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:25.344033 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:25.410018 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 123 similar messages]
I20240503 08:17:25.546386 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:25.558773 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling FlushMRSOp(72a949e76ab5477680a4f9912e9672e9): perf score=0.033345
I20240503 08:17:25.569219 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:25.571897 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:17:25.574362 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:25.588474 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:25.591897 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:25.611335 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:25.631518 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:25.687059 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: FlushMRSOp(72a949e76ab5477680a4f9912e9672e9) complete. Timing: real 0.128s	user 0.053s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":23601,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":3555,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9887,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":36004,"threads_started":2,"wal-append.queue_time_us":334}
I20240503 08:17:25.688567 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(72a949e76ab5477680a4f9912e9672e9): 4583 bytes on disk
I20240503 08:17:25.689246 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(72a949e76ab5477680a4f9912e9672e9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 08:17:25.693126 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:25.701675 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.747584 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:25.767452 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:25.808820 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:25.809568 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:25.818063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:25.846544 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:25.874682 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:25.878150 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:25.895308 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:25.915299 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:25.947487 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:25.999070 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:26.064560 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:26.209477 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:26.257660 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:17:26.293097 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:26.303123 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:26.311508 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:26.322682 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:26.326434 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:26.327145 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:26.327972 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:26.367763 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:26.385465 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:17:26.514843 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:26.565797 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:26.566874 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:26.570451 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:26.592598 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:26.593437 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:26.593482 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:26.633955 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:26.660099 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:26.709131 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(dc9575f64eb741948e51ccf06f48228a): perf score=0.033357
W20240503 08:17:26.742250 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:26.753949 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(dc9575f64eb741948e51ccf06f48228a) complete. Timing: real 0.045s	user 0.041s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":1779,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":834,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":571,"threads_started":2,"wal-append.queue_time_us":407}
I20240503 08:17:26.755532 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(dc9575f64eb741948e51ccf06f48228a): 4836 bytes on disk
I20240503 08:17:26.756183 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(dc9575f64eb741948e51ccf06f48228a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 08:17:26.820514 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:26.831321 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:26.835943 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:26.836999 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:26.857492 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:26.869267 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:26.900810 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:26.904695 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:26.908735 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:26.928490 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:27.048859 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:27.292994 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:27.298585 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:27.316540 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:27.317020 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:27.320333 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:27.323716 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:27.338362 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:27.357240 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(4e86f4f3a01048e69f553c6557d79b71): perf score=0.033388
I20240503 08:17:27.372442 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:27.376521 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:27.414525 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:27.417634 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling FlushMRSOp(f7a5221747794a13aa4ed763006ba291): perf score=0.033335
I20240503 08:17:27.422202 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(4e86f4f3a01048e69f553c6557d79b71) complete. Timing: real 0.065s	user 0.040s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":394,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":2102,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":508,"threads_started":2,"wal-append.queue_time_us":380}
I20240503 08:17:27.423472 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(4e86f4f3a01048e69f553c6557d79b71): 4923 bytes on disk
I20240503 08:17:27.424117 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(4e86f4f3a01048e69f553c6557d79b71) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240503 08:17:27.460177 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: FlushMRSOp(f7a5221747794a13aa4ed763006ba291) complete. Timing: real 0.042s	user 0.029s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":966,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":894,"threads_started":2,"wal-append.queue_time_us":738}
I20240503 08:17:27.461700 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(f7a5221747794a13aa4ed763006ba291): 4923 bytes on disk
I20240503 08:17:27.462430 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(f7a5221747794a13aa4ed763006ba291) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240503 08:17:27.476063 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:27.558501 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:27.624111 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:27.658119 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:27.658674 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:27.663173 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:27.665441 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:27.683224 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:27.684751 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:27.699807 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:27.742789 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:27.759563 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:27.773101 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:27.885462 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:27.909967 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:27.924535 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:27.930063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:27.931510 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:27.948963 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:27.994382 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:28.027571 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:28.123172 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:28.138720 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:28.163810 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:28.280320 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:17:28.283880 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:28.333355 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:28.370097 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:28.404824 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:28.413327 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:28.423955 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:28.427809 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:28.443974 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:28.454892 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:28.472381 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:28.541848 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:28.644588 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:17:28.652364 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:28.658807 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:28.666325 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:28.677807 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:28.689841 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(f65ab6dba1124573962e2da58aa1859c): perf score=0.033373
I20240503 08:17:28.690953 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:28.694406 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:28.766305 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:28.784081 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(f65ab6dba1124573962e2da58aa1859c) complete. Timing: real 0.094s	user 0.038s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":462,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1551,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":926,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7773,"threads_started":2,"wal-append.queue_time_us":2457}
I20240503 08:17:28.785589 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(f65ab6dba1124573962e2da58aa1859c): 4583 bytes on disk
I20240503 08:17:28.786302 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(f65ab6dba1124573962e2da58aa1859c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240503 08:17:28.805053 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:28.810599 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:28.901922 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:28.935846 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:28.945940 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:28.985675 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:28.987044 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:28.988436 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:29.041735 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:29.042071 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:29.052426 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:29.076068 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:29.241911 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:29.301221 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:29.325033 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:29.353513 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:29.357432 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:29.369952 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:29.422407 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:29.438644 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:29.448406 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:29.449678 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:29.457332 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:29.487807 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:29.644129 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:29.687094 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(5d16c2981213464cbefa44b6ca509182): perf score=0.033354
I20240503 08:17:29.697866 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:29.714562 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:29.721400 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:29.752038 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:29.761097 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:29.764468 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(5d16c2981213464cbefa44b6ca509182) complete. Timing: real 0.077s	user 0.041s	sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6452,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":5496,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11478,"threads_started":2,"wal-append.queue_time_us":4760}
I20240503 08:17:29.768386 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(5d16c2981213464cbefa44b6ca509182): 4924 bytes on disk
I20240503 08:17:29.776619 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(5d16c2981213464cbefa44b6ca509182) complete. Timing: real 0.004s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 08:17:29.784994 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:29.787705 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:29.851383 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:29.932015 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:29.996294 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:29.997620 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:30.019294 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:30.022238 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:30.026517 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:30.041656 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:30.041994 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:30.051699 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:30.085263 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:30.091524 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:30.203526 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:30.272171 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(4e86f4f3a01048e69f553c6557d79b71): perf score=0.033368
W20240503 08:17:30.279489 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:30.280124 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling FlushMRSOp(72a949e76ab5477680a4f9912e9672e9): perf score=0.033393
I20240503 08:17:30.293872 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(dc9575f64eb741948e51ccf06f48228a): perf score=0.033394
W20240503 08:17:30.332502 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:30.345314 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: FlushMRSOp(72a949e76ab5477680a4f9912e9672e9) complete. Timing: real 0.065s	user 0.050s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4203,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1582,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":857,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21083,"threads_started":2,"wal-append.queue_time_us":17099}
I20240503 08:17:30.346956 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(72a949e76ab5477680a4f9912e9672e9): 4583 bytes on disk
I20240503 08:17:30.347693 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(72a949e76ab5477680a4f9912e9672e9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 08:17:30.353567 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(4e86f4f3a01048e69f553c6557d79b71) complete. Timing: real 0.081s	user 0.058s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6990,"dirs.run_cpu_time_us":279,"dirs.run_wall_time_us":13154,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9296,"threads_started":2,"wal-append.queue_time_us":2500}
I20240503 08:17:30.355134 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(4e86f4f3a01048e69f553c6557d79b71): 4923 bytes on disk
I20240503 08:17:30.355687 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(4e86f4f3a01048e69f553c6557d79b71) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:17:30.371991 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(dc9575f64eb741948e51ccf06f48228a) complete. Timing: real 0.078s	user 0.059s	sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":412,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1052,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":896,"thread_start_us":701,"threads_started":2,"wal-append.queue_time_us":359}
I20240503 08:17:30.373353 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(dc9575f64eb741948e51ccf06f48228a): 4836 bytes on disk
I20240503 08:17:30.374960 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(dc9575f64eb741948e51ccf06f48228a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":183,"lbm_reads_lt_1ms":4}
I20240503 08:17:30.375874 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling FlushMRSOp(55067e08fb694e90b6366d41cc962095): perf score=0.033388
W20240503 08:17:30.398475 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:30.412940 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: FlushMRSOp(55067e08fb694e90b6366d41cc962095) complete. Timing: real 0.037s	user 0.030s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1562,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":326,"threads_started":1,"wal-append.queue_time_us":422}
I20240503 08:17:30.414094 14797 maintenance_manager.cc:419] P c0fd1ebec9da48d4b0c1ae0f0ee96452: Scheduling UndoDeltaBlockGCOp(55067e08fb694e90b6366d41cc962095): 4583 bytes on disk
I20240503 08:17:30.414593 14688 maintenance_manager.cc:643] P c0fd1ebec9da48d4b0c1ae0f0ee96452: UndoDeltaBlockGCOp(55067e08fb694e90b6366d41cc962095) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 08:17:30.427739 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240503 08:17:30.487963 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:30.490188 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:30.497924 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:30.516969 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:30.517288 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:30.522584 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:30.522889 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:30.538264 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:30.650388 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:30.732985 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:30.739521 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:30.758651 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:30.758703 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:30.766219 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:30.777907 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240503 08:17:30.808569 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:30.809195 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:30.864305 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:30.906548 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:30.950790 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:30.963796 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:17:31.036495 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:31.039954 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:31.046237 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:31.062878 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:31.068663 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:31.084435 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:31.260453 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:17:31.266698 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:31.270785 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:31.370208 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:31.388453 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:17:31.389432 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:31.446286 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:31.482990 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:31.488579 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:31.494599 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:31.502442 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:31.506471 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:31.526010 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:31.537894 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:31.611795 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(dd11460051ab40febc2fe6aff58b5028): perf score=0.033389
I20240503 08:17:31.652894 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(dd11460051ab40febc2fe6aff58b5028) complete. Timing: real 0.041s	user 0.029s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1930,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1372,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1738,"threads_started":2,"wal-append.queue_time_us":1579}
I20240503 08:17:31.654060 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(dd11460051ab40febc2fe6aff58b5028): 4838 bytes on disk
I20240503 08:17:31.654767 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(dd11460051ab40febc2fe6aff58b5028) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240503 08:17:31.656193 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(169bd9ea6772420abf757dd55cf44d31): perf score=0.033388
W20240503 08:17:31.687678 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:31.716059 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:31.727552 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(169bd9ea6772420abf757dd55cf44d31) complete. Timing: real 0.071s	user 0.053s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1629,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":809,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5817,"threads_started":1,"wal-append.queue_time_us":5962}
I20240503 08:17:31.728933 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(169bd9ea6772420abf757dd55cf44d31): 4923 bytes on disk
I20240503 08:17:31.729516 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(169bd9ea6772420abf757dd55cf44d31) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240503 08:17:31.776590 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:31.802034 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:31.802759 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:31.816435 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:31.816597 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:31.829023 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:31.832648 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:31.857671 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:31.860263 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:31.884649 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:31.952919 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:31.982355 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(c130eb7ea8a74f4b8d9b79e23af70067): perf score=0.033386
I20240503 08:17:32.018919 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(c130eb7ea8a74f4b8d9b79e23af70067) complete. Timing: real 0.036s	user 0.034s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1646,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":359,"threads_started":1,"wal-append.queue_time_us":463}
I20240503 08:17:32.020364 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(c130eb7ea8a74f4b8d9b79e23af70067): 4838 bytes on disk
W20240503 08:17:32.020804 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:32.021298 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(c130eb7ea8a74f4b8d9b79e23af70067) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":320,"lbm_reads_lt_1ms":4}
I20240503 08:17:32.112920 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:32.113732 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:32.129516 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:32.129947 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:32.137640 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:32.151947 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:32.242645 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:32.273514 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling FlushMRSOp(da1f165b4a5647a48ad7ff951fb3fbf6): perf score=0.033379
W20240503 08:17:32.275678 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:32.278396 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:32.321056 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: FlushMRSOp(da1f165b4a5647a48ad7ff951fb3fbf6) complete. Timing: real 0.047s	user 0.036s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1640,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":284,"threads_started":1,"wal-append.queue_time_us":392}
I20240503 08:17:32.322295 14667 maintenance_manager.cc:419] P 0d77c4b1f12049dea71b97e86be2b288: Scheduling UndoDeltaBlockGCOp(da1f165b4a5647a48ad7ff951fb3fbf6): 4583 bytes on disk
I20240503 08:17:32.322865 14558 maintenance_manager.cc:643] P 0d77c4b1f12049dea71b97e86be2b288: UndoDeltaBlockGCOp(da1f165b4a5647a48ad7ff951fb3fbf6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 08:17:32.423067 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:32.474520 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:32.553426 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:32.562304 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:32.569100 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:32.573621 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:32.576632 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:32.601603 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:32.606956 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:32.611975 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:32.624226 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:32.769513 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:32.785486 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:32.826771 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:32.834239 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:32.851966 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:32.861892 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:32.868167 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:32.876963 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:32.890997 14817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.129:37829, user_credentials={real_user=slave}} blocked reactor thread for 83437.5us
I20240503 08:17:32.903280 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:32.928946 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:32.967064 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:33.156446 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:33.156935 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:33.164470 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:33.164789 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:33.176066 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:33.198457 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:33.205806 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:33.211560 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:33.218886 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:33.275651 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:33.334878 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:33.370867 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:33.433746 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:17:33.619341 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:33.619341 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:33.621363 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:33.625032 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:33.634443 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:33.636847 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:33.668238 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:17:33.689321 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:33.788223 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:33.851585 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:33.852856 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:33.854804 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:33.857108 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:33.870098 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:33.888974 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:33.905414 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:33.957530 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:33.962589 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.009670 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.029374 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.054693 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:34.114279 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:34.126706 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:34.138599 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:34.144487 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:34.146773 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:34.150529 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:34.168401 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:34.200853 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.305701 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:34.452982 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.479786 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.499079 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:34.572046 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:34.588078 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:34.598106 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:34.610944 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:34.653574 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:34.671510 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:34.685838 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:34.688076 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:34.730612 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:34.784588 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:34.866839 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:34.870115 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:34.886938 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:34.891189 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:34.899221 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:34.906327 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:34.921330 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:34.983119 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:35.036782 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:35.101465 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:35.148262 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:35.152689 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:35.153882 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:35.172196 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:35.173063 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:35.177400 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:35.209997 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:35.262948 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:35.484112 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:35.501899 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:17:35.534801 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:35.598268 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:35.612596 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:35.616041 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:35.620637 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:17:35.632696 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:35.635057 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:35.637728 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:35.654408 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:35.657016 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:35.671094 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:35.694892 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:35.784991 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:17:35.792155 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:35.844287 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:35.859405 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:35.860337 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:35.887154 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:35.890036 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:35.915336 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:35.918082 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:17:36.040289 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:36.120290 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:36.139482 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:36.147903 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:36.156780 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:36.170605 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:36.171516 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:36.181748 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:36.204145 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:36.394376 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:17:36.521914 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:36.521888 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:36.533970 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:36.536760 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:36.551561 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:36.572767 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:36.587816 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:36.592308 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:36.604835 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:36.623236 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:36.642621 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:36.644407 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:36.653723 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:36.686894 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:36.711573 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:36.718030 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:36.888222 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:36.889758 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:36.890170 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:36.893919 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:36.903102 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:36.919582 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:36.992817 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:37.121773 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:37.131534 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:37.153259 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:37.170328 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:37.171154 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:37.172133 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:37.183800 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:37.186408 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:37.203187 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:37.302605 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:37.322094 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:37.433097 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:37.435461 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:37.522998 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:37.559893 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:37.619068 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:37.627447 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:37.643946 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:37.646055 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:37.680508 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:37.683532 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:37.748775 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:37.752753 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:37.755426 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:37.958547 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:37.965425 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:37.986361 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:37.988517 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:38.013867 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:38.035281 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:38.060515 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:38.077065 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:38.090667 14425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.135:46299, user_credentials={real_user=slave}} blocked reactor thread for 125275us
W20240503 08:17:38.097407 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:38.107882 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:38.129595 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:38.130348 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:38.146206 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:17:38.284798 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:38.286671 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:38.295979 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:38.312824 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:38.314679 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:38.315814 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:38.365902 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:17:38.480456 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:38.481382 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:38.547194 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:38.602762 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:38.702095 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:38.731549 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:38.743170 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:38.747665 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:38.752907 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:38.774076 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:38.796684 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:38.851133 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:38.943861 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:38.979046 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:38.980528 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:38.993150 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:39.002651 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:39.020080 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:39.024407 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:39.045961 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:39.055123 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:39.172646 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:39.194315 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:39.238466 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:39.242230 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:39.243036 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:39.247061 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:39.257954 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:39.279296 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:39.289361 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:39.294427 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:39.298364 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:39.373551 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:39.404266 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:39.478261 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:39.563107 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:39.635555 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:39.690765 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:39.695900 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:39.702988 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:39.703981 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:39.724037 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:39.740377 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:39.833587 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:39.924754 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:39.953874 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:39.964401 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:39.965605 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:39.965994 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:39.986277 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:39.996421 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:40.043195 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:40.047073 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:40.092156 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:40.157047 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:40.177109 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:40.221127 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:40.242193 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:40.250862 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:40.259476 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:40.261451 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:40.275213 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:40.304216 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:40.304865 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:40.386693 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:40.450467 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:40.470954 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:40.559708 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:40.607986 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:40.675995 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:17:40.679266 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:40.695968 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:40.702661 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:40.706691 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:40.718876 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:40.739794 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:40.746541 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:40.758683 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:17:40.762440 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:40.776530 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:17:40.905207 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:17:40.990204 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:41.005874 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:41.016887 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:41.020875 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:41.060299 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:41.088587 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:41.089717 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:41.147835 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:41.149155 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:41.258674 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:41.264865 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:41.266386 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:41.281493 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:41.302852 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:41.309563 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:41.310854 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:41.412447 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:41.477727 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:41.566962 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:41.578344 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:41.602317 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:17:41.603250 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:41.673492 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:41.674971 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:41.724372 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:41.731968 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:41.732852 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:41.752375 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:41.761873 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:41.764937 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:41.770740 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:41.820420 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:41.849499 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:41.998739 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:42.005183 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:42.029156 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:42.043455 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:42.067936 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:42.097914 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:42.100731 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:42.148304 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:42.184463 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:42.187734 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:42.287791 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:42.307067 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:42.313695 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:42.314558 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:42.320144 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:42.331442 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:42.381418 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:42.572319 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:42.612504 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:42.626389 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:42.648269 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:42.721752 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:42.742383 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:42.742497 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:42.745940 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:42.763083 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:42.786553 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:42.792140 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:42.794193 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:42.844089 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:42.916978 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:43.021081 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:17:43.025380 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:43.035718 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:43.036068 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:43.037683 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:43.050467 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:43.055662 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:43.102082 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:43.138139 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:43.172158 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:43.182346 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:43.182569 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:17:43.203511 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:43.225803 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:43.257803 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:43.263603 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:43.275827 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:43.283936 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:43.312568 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:43.317193 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:43.573722 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:43.574566 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:43.684000 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:43.706635 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:43.750828 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:43.757464 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:43.762425 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:43.767539 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:43.785265 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:43.845405 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:43.851389 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:43.962754 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:44.056288 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:44.061652 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:44.064980 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:44.078523 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:44.085314 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:44.099051 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:44.107661 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:44.107940 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:44.154667 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:44.175769 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:44.238756 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:44.282722 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:44.299623 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:44.312373 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:44.322552 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:44.330509 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:44.345382 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:44.345809 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:44.362493 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:44.394313 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:44.573688 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:44.656021 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:44.763311 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:44.765041 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:44.766383 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:44.775789 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:44.777194 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:44.803162 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:44.805786 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:44.867182 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:44.890128 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:44.957518 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:45.046708 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:45.073503 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:45.074532 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:45.080318 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:45.095490 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:45.105762 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:45.107673 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:45.191447 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:45.195036 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:45.201759 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:45.306650 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:45.318470 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:45.325806 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:45.329053 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:45.330147 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:45.332418 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:45.352396 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:45.404146 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:45.471935 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:45.521100 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:45.611064 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:45.626947 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:45.629644 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:45.633270 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:45.706974 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:17:45.725908 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:45.743425 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:45.751629 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:45.780920 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:45.791998 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:17:45.793314 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:45.803306 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:45.814985 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:45.837407 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:17:45.919186 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:45.942332 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:17:46.042963 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:46.075278 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:46.079399 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:46.097363 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:46.097767 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:46.104954 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:46.126205 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:46.189690 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:46.212055 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:17:46.339143 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:46.344203 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:17:46.349005 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:46.361132 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:46.374023 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:46.374558 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:46.410506 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:46.429472 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:46.460744 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:46.582557 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:46.675891 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:46.711328 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:46.765292 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:46.766507 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:46.793462 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:46.817076 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:46.827214 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:46.834786 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:46.835645 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:46.850750 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:46.854696 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:47.004101 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:47.086064 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:17:47.087193 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:47.101584 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:47.103729 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:47.105551 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:47.117084 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:47.138022 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:47.153895 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:47.167752 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:47.290935 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:47.340911 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:47.355499 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:47.369679 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:47.390223 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:47.401247 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:47.425472 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:47.469750 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:47.539139 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:47.685253 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:47.686625 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:47.698745 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:47.709249 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:47.718292 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:47.763518 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:47.786638 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:47.825966 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:47.839555 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:47.880563 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:47.884963 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:47.914659 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:47.941147 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:47.947399 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:48.025887 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:48.048514 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:17:48.137650 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:48.156996 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:48.166618 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:17:48.171053 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:48.181282 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:48.208348 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:48.230129 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:48.247982 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:48.317580 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:48.347491 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:48.413023 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:48.443055 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:48.444809 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:48.456348 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:48.485332 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:48.509912 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:48.517427 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:48.531325 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:48.538594 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:17:48.659760 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:48.767645 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:48.773062 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:48.900365 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:48.907574 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:48.917029 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:48.933436 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:48.951743 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:17:48.953043 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:48.968678 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:48.985296 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:49.049392 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:49.159369 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:49.164633 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:49.217805 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:49.243934 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:49.245013 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:49.254695 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:49.265132 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:49.275657 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:49.289264 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:49.314465 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:49.327623 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:49.354645 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:49.503427 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:49.506983 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:49.531358 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:49.537142 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:49.539592 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:49.542953 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:49.546766 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:49.585584 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:49.615113 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:49.771605 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:49.967361 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:50.009083 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:50.047224 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:50.053334 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:50.053930 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:50.060135 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:50.067060 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:50.078871 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:50.078924 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:50.090085 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:50.166307 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:50.192054 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:50.212687 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:50.236039 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:50.306638 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:50.310539 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:50.312160 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:50.329121 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:50.333130 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:50.337663 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:50.363502 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:50.408389 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:50.559986 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:50.560438 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:50.562559 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:50.570827 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:17:50.583166 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:50.587471 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:50.592944 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:50.618444 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:50.621217 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:50.644431 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:50.695801 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:50.748999 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 08:17:50.761628 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:50.954071 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:17:50.955348 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:51.012578 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:51.013074 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:17:51.041822 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:51.055697 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:51.072067 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:51.080492 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:51.084332 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:51.104300 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:51.123817 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:51.153316 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:51.278611 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:51.301079 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:51.313596 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:51.323081 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:51.337281 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:51.370131 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:51.374997 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:51.379957 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:51.380563 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:51.484858 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:51.529796 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:51.571717 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:51.573320 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:51.597201 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:51.623015 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:51.668431 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:51.690585 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:51.700119 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:51.702872 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:51.802875 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:51.884605 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:51.892150 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:51.911242 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:51.948608 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:52.094753 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:52.098702 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:52.113704 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:52.114877 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:52.115363 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:52.117125 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:52.165755 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:52.183300 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:52.334028 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:52.347185 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:52.349072 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:52.352891 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:52.367064 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:52.390281 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:52.394780 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:52.397445 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:52.401995 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:52.516614 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:52.587494 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:52.602587 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:52.605146 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:52.619359 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:52.645712 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:52.673911 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:52.676736 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:52.682468 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:52.824759 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:52.834059 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:52.874758 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:52.908833 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:52.917474 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:52.944413 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:52.951692 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:52.978737 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:53.067682 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:53.082701 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:53.104633 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:53.105724 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:53.110363 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:53.115476 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:53.122551 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:53.220640 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:53.373131 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:53.373551 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:53.373864 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:53.383651 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:53.404551 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:53.412667 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:53.432883 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:53.491426 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:53.514351 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:53.582387 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:53.592133 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:17:53.637374 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:53.642408 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:53.643546 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:53.646489 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:53.667151 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:53.668169 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:53.683854 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:17:53.871932 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:53.897609 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:53.966675 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:53.980171 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:53.981139 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:54.039506 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:17:54.098553 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:54.104696 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:54.115167 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:54.115232 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:54.131222 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:54.137786 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:54.218639 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:54.299328 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:54.331646 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:17:54.332218 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:54.344344 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:54.351788 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:54.355777 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:54.386777 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:54.389814 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:54.389804 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:54.505609 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:54.526726 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:54.531435 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:54.590282 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:54.596419 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:54.604990 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:54.647325 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:54.669317 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:54.670020 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:54.683192 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:54.728596 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:54.842126 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:54.892856 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:54.893663 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:54.983541 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:55.046116 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:55.058470 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:55.069731 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:55.077873 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:55.097354 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:55.120812 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:55.126938 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:55.147182 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:55.231232 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:55.308050 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:55.322656 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:17:55.323252 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:55.323789 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:17:55.344899 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:55.371057 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:55.399343 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:55.405763 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:55.408891 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:55.426787 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:55.457859 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:55.611790 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:55.613926 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:55.626935 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:55.632946 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:55.641480 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:55.645998 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:55.654301 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:55.656085 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:55.702133 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:55.782709 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:17:55.856779 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:55.933182 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:56.067282 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:56.094189 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:56.106909 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:17:56.108263 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:56.112905 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:56.120196 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:56.124027 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:56.131374 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:56.151741 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:56.161229 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:56.197229 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:56.231359 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:17:56.358278 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:56.366647 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:56.399524 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:56.419198 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:56.420176 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:56.454407 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:56.461233 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:56.487013 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:56.535521 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:56.613926 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:56.674575 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:56.688676 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:56.689159 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:56.692045 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:56.692942 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:56.701879 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:56.736426 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:56.737004 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:56.864027 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:17:56.865069 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:56.933241 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:56.940043 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:57.047217 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:57.113165 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:57.165463 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:57.172087 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:57.178797 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:57.182942 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:17:57.186259 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:57.192193 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:57.198007 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:57.206784 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:57.312310 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:57.360714 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:57.413136 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:57.421216 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:57.431581 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:57.433568 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:57.435741 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:57.436834 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:57.469087 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:57.479895 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:57.696384 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:57.703558 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:57.709890 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:57.716125 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:57.725734 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:57.758836 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:57.766148 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:57.787071 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:57.814121 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:57.829764 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:57.838912 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:57.840673 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:57.846087 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:57.943527 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:58.128919 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:58.184556 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:58.187603 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:58.188191 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:58.190433 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:58.208978 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:58.215632 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:58.227643 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:17:58.320673 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:58.346982 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:58.389465 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:58.414734 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:17:58.416589 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:17:58.429204 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:58.429811 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:58.453178 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:58.457583 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:58.482434 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:58.539700 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:58.651017 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:17:58.671651 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:17:58.673213 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:58.680406 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:58.689006 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:58.690030 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:58.729667 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:58.733057 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:58.828696 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:58.923213 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:58.996934 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:17:58.997704 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:17:59.055249 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:59.090871 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:17:59.152442 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:59.158733 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:59.161697 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:59.185448 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:59.207466 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:17:59.212175 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:17:59.213881 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:59.321992 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:17:59.413098 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:59.418263 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:59.420480 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:59.429497 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:59.434096 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:59.436056 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:17:59.471391 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:59.521910 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:59.555228 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:59.625105 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:59.637732 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:59.654793 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:17:59.657671 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:17:59.658586 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:17:59.664492 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:17:59.677992 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:17:59.689827 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:17:59.697880 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:17:59.726603 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:17:59.873230 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:59.953637 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:17:59.975026 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:00.036607 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:00.116482 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:00.131410 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:00.132468 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:00.139000 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:00.163504 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:00.171419 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:00.353935 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:00.359109 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:00.373323 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:00.379102 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:00.380895 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:00.385991 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:00.398641 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:00.409920 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:00.416255 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:00.432713 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:00.441726 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:00.441879 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:00.460297 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:00.470829 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:00.503690 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:18:00.622865 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:00.633030 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:00.638199 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:00.651705 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:00.662635 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:18:00.665889 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:00.672721 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:00.754118 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:00.839669 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 08:18:00.840502 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:00.855655 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:00.883500 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:00.988009 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:01.087235 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:01.113111 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:01.124507 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:01.127311 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:18:01.130507 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:01.160135 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:01.160737 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:01.213858 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:01.263676 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:01.310043 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:18:01.312570 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:01.337505 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:01.358976 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:01.362773 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:01.385797 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:01.385958 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:01.393546 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:01.393708 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:01.574790 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:01.610428 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:01.626063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:01.632602 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:01.638412 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:01.642186 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:01.665937 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:01.672559 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:01.690821 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:01.806613 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:01.891795 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:01.917774 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:02.009580 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:02.051661 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:18:02.084502 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:02.089751 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:02.096666 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:02.105437 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:02.114727 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:02.129817 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:02.131652 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:02.149766 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:02.180723 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:02.198001 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:02.348565 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:02.349570 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:02.363718 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:02.364509 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:02.381649 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:02.387499 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:02.414779 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:02.435892 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:02.437173 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:02.459178 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:02.602352 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:02.618660 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:02.623862 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:02.632393 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:02.644553 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:02.646942 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:02.652722 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:02.726467 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:02.794137 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:02.881325 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:02.977572 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:03.037468 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:03.060550 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:03.083140 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:03.095054 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:03.095209 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:03.100504 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:03.106673 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:03.114811 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:03.117254 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:18:03.132335 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:03.191512 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:03.217998 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:03.261585 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:03.360973 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:03.362809 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:03.373720 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:03.377660 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:03.382405 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:03.399596 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:03.408911 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:03.413586 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:03.606287 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:03.623411 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:03.640723 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:03.641777 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:03.649510 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:03.653899 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:03.657584 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:03.685443 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:03.695791 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:03.729166 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:03.755539 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:18:04.022917 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:18:04.096622 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:04.097522 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:04.114621 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:18:04.116020 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:04.116581 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:04.131021 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:04.134207 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:04.152588 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:04.156719 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:04.174357 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:04.190786 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:04.192087 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:04.339152 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:18:04.368916 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:04.382602 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:04.392441 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:04.416478 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:04.423260 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:04.433686 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:04.626225 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:04.643911 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:04.662564 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:04.682327 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:04.700853 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:04.711999 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:04.712716 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:04.730588 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:04.730978 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:04.749243 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:04.779224 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:04.810401 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:04.832412 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:04.965837 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:05.040411 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:05.056814 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:05.132520 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:05.137543 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:05.137962 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:05.142288 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:05.147001 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:05.175293 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:05.175316 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:05.217511 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:05.392019 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:05.395579 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:05.399158 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:05.402992 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:05.407891 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:05.434677 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:05.460192 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:05.515440 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:05.516386 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:05.550772 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:05.552202 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:05.572732 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:05.614418 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:05.629436 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:18:05.642459 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:05.645532 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:05.646549 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:05.662547 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:05.663612 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:18:05.677557 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:05.680404 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:05.688271 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:05.909590 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:18:05.954371 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:05.996613 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:06.026304 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:06.089442 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:06.099205 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:06.108122 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:06.129848 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:06.134198 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:06.155488 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:06.197621 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:06.224573 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:18:06.282521 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:06.341370 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:06.341400 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:06.345930 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:06.357887 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:06.365155 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:06.371820 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:06.382683 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:06.464603 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:18:06.534772 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:06.578868 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:06.618952 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:06.619345 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:06.619439 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:06.632388 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:06.692924 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:06.695379 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:06.819671 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:06.859344 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:06.876231 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:06.904809 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:06.972126 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:06.992246 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:07.064895 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:07.071285 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 08:18:07.103917 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:07.109460 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:07.112607 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:07.119201 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:07.124207 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:07.151834 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:07.190562 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:07.275156 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:07.279485 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:07.287480 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:07.302634 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:18:07.345038 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:07.361546 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:07.363623 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:07.363618 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:07.385052 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:07.393339 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:07.397517 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:07.471843 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:07.594403 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:07.608287 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:07.615866 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:07.630615 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:07.653244 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:07.661056 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:18:07.667001 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:07.782886 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:07.878353 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:07.949239 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:08.057207 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:18:08.064460 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:08.070726 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:08.081610 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:18:08.084302 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:08.091061 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:08.091503 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:08.092295 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:08.102227 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:08.160327 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:08.160476 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:18:08.161502 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:08.235617 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:08.260946 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:08.282547 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:08.349330 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:08.356580 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:08.364082 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:08.401947 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:08.408061 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:08.420835 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:08.605959 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:08.618755 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:08.631512 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:08.640945 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:08.641201 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:08.678186 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:08.691093 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:08.698578 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:08.815909 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:08.850013 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:08.884614 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:08.910002 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:18:09.017866 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:09.131431 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:09.137940 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:09.141112 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:09.153215 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:09.157711 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:09.174077 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:09.182083 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:09.262079 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:09.336372 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:09.374905 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:09.378901 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:09.387676 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:09.396309 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:09.411931 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:09.419519 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:09.445669 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:09.447328 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:09.448172 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:18:09.449126 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:09.493867 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:09.600435 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:09.625250 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:09.634692 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:09.647281 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:09.659355 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:09.664319 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:09.665693 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:09.690375 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:09.701900 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:09.849473 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:09.857131 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:09.858258 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:09.872335 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:09.986482 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:10.042014 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:10.100965 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:10.112850 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:10.120239 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:10.125402 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:10.150390 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:10.162667 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:10.257233 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:10.357666 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:10.359833 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:10.363601 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:10.378808 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:10.381071 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:10.384496 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:10.402509 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:10.414206 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:10.429087 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:10.480542 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:10.564998 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:10.609117 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:10.626014 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:10.626358 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:10.633324 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:10.645160 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:10.645797 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:10.646967 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:10.694803 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:10.724550 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:10.768671 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:10.791682 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:10.904209 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:11.017133 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:11.038589 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 122 similar messages]
I20240503 08:18:11.066047 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:11.085470 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:11.093160 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:11.093588 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:11.118058 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:11.132742 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:11.190320 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:11.191284 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:18:11.281607 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:11.299139 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:18:11.300117 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:18:11.359079 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:11.385769 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:11.386435 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:11.388391 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:11.402269 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:11.408594 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:11.422564 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:11.570749 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:11.645347 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:11.673035 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:11.675657 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:11.680316 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:11.686856 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:11.702550 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:11.724193 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:11.749778 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:11.750931 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:11.828298 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:11.831307 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:12.005165 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.147475 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 08:18:12.154134 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:12.163554 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:12.166649 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:12.167910 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:12.177470 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:12.178448 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:12.180718 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:12.201833 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:12.267591 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.313547 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:12.408962 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:12.412715 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:12.425415 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:12.425769 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:12.427835 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:12.452510 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:12.456727 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:12.518388 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.561477 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.604732 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.642349 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:12.653328 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:12.663794 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.664186 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:12.683912 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:12.691859 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:12.712612 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:12.735735 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:12.744640 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:12.752887 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:12.939462 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:13.092986 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:13.095234 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:13.146826 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:13.149111 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:18:13.152004 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:13.156469 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:13.175316 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:13.176589 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:13.180243 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:13.222086 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:13.241586 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:13.257154 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:13.294255 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:13.298878 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:18:13.363595 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:13.386746 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:13.403396 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:13.403908 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:13.407296 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:13.427884 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:13.428920 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:13.528586 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:13.618649 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:13.671010 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:13.673864 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:13.681783 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:13.686822 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:13.693809 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:13.699391 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:13.720224 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:18:13.782018 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:13.782963 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:14.002318 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:14.182745 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:14.198472 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:14.206969 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:14.210638 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:18:14.221246 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:14.231237 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:14.235285 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:14.252648 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:14.254715 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:14.259057 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:14.284994 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:14.367434 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:18:14.444607 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:14.449144 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:14.463220 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:14.464421 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:14.478444 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:14.481729 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:14.597010 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:14.598130 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:14.692334 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:14.702515 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:14.708467 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:18:14.725109 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:14.726810 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:14.732791 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:14.734696 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:14.747949 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:14.782143 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:14.844295 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:14.844755 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.011790 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.064770 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.122236 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:15.148862 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:15.152494 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:15.153520 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:15.156668 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:15.178807 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.185304 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:18:15.189188 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:15.189486 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:15.190431 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:15.329759 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:15.399425 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:15.401877 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:15.402546 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:15.408134 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:15.430698 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:15.442464 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:15.518400 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:15.590687 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:15.652784 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:15.665556 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:15.668411 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:15.673249 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:15.681233 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:15.704371 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:15.724954 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:15.738710 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:15.757593 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.815207 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.928547 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.934374 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:15.934763 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:15.947247 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:15.967270 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:16.022238 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:18:16.042651 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 118 similar messages]
I20240503 08:18:16.138705 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:16.139441 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:16.139444 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:16.147606 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:16.154125 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:16.182308 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:16.185760 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:16.338613 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:18:16.391180 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:16.406448 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:16.413053 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:16.413058 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:16.415063 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:16.416114 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:16.418282 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:16.442606 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:16.513676 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:16.664733 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:16.680296 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:16.681231 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:16.681380 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:16.683240 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:16.703276 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:16.707351 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:16.744277 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:16.794821 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:16.819964 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:16.848930 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:16.895594 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:16.930617 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:17.093478 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:17.106666 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:17.117113 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:17.127911 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:17.154433 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:17.155582 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:17.160327 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:17.175325 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:17.215200 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:18:17.225982 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:17.387172 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:17.398290 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:17.411116 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:17.411625 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:17.425190 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:17.442068 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:17.451073 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:17.556350 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:17.611547 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:17.629249 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:17.650111 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:17.665483 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:17.667407 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:17.670581 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:17.688870 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:17.711272 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:17.724347 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:17.724680 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:17.741075 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:17.770195 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:17.820381 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:17.995393 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:18.065541 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:18.119071 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:18.160410 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:18.185201 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:18.192132 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:18.209513 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:18.216966 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:18.237306 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:18.251968 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:18.265241 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:18.290860 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:18.393007 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:18.441980 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:18.451951 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:18.453729 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:18.459674 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:18.463083 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:18.508347 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:18.513113 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:18.600561 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:18.614509 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:18.620299 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:18:18.690357 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:18:18.710880 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:18.725831 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:18.729091 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:18.733897 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:18.734444 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:18.784597 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:18.787734 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:18.816025 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:18.945531 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:19.063445 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:19.086443 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:19.168251 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:19.174345 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:19.193852 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:19.206461 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:19.214004 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:19.215688 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:19.216342 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:19.322439 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:19.337862 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:19.390998 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:19.412986 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:19.427987 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:19.429821 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:19.443665 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:19.479025 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:19.489281 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:19.498535 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:19.526250 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:19.617329 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:18:19.732010 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:19.738723 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:19.739619 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:19.746455 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:19.768608 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:19.770709 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:19.795856 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:19.797662 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:19.805431 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:19.868820 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:19.912662 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:20.004138 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:20.045657 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:20.072413 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:20.186285 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:20.195327 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:20.200371 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:20.216656 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:20.235963 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:20.257084 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:20.260707 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:20.271994 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:20.346973 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:20.357128 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:20.377878 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:20.444717 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:20.460592 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:20.463124 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:20.489809 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:20.490214 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:20.505707 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:20.605876 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:20.672053 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:20.701608 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:20.710116 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:20.720075 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:20.735216 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:20.750366 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:20.795096 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:20.797004 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:20.821996 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:20.826071 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:20.955132 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:21.082911 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:18:21.083777 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:21.094035 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:18:21.198094 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:21.209055 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:21.222198 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:21.223754 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:21.225567 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:21.231153 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:18:21.243932 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:21.263402 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:21.279687 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:21.296353 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:21.398320 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:21.486768 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:21.501590 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:21.502945 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:21.507808 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:21.554632 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:21.567729 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:21.580013 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:18:21.600586 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:21.638777 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:21.671764 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:21.748521 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:21.761492 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:21.781838 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:21.789484 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:21.798978 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:21.803551 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:21.812316 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:21.816941 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:21.976188 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:21.983922 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:22.084427 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:22.152015 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:22.199749 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:22.207612 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:22.212087 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:22.233376 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:22.250653 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:18:22.251860 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:22.252405 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:22.279927 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:22.281365 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:22.283632 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:22.308074 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:18:22.325081 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:22.329443 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:22.541064 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:22.553350 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:22.554076 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:22.571635 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:22.580950 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:22.590691 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:22.602350 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:22.626458 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:22.663789 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:22.734759 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:22.797096 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:18:22.804854 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:22.820624 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:22.826187 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:22.834970 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:22.916905 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:22.917271 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:22.922022 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:22.956393 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:23.028592 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:23.059526 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:23.070354 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:23.197007 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:23.222054 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:23.297755 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:23.305088 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:23.309952 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:23.314479 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:23.336047 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:23.362003 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:23.384255 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:23.475142 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:18:23.486814 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:23.497294 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:18:23.562251 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:23.569541 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:23.575948 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:23.597779 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:23.600661 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:23.601316 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:23.624863 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:23.672602 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:23.781133 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:18:23.797786 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:23.802855 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:23.812850 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:23.833177 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:23.833837 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:23.845718 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:23.847923 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:23.893996 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:24.107941 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:24.162272 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:24.163956 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:24.245834 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:24.251585 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:24.259821 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:24.264108 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:24.269102 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:24.284950 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:24.328421 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:24.349377 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:24.364400 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:24.376884 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:24.519050 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:24.520233 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:24.530478 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:24.535060 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:24.571174 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:24.577953 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:24.582873 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:24.588608 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:24.652896 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:24.747767 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:24.761382 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:24.765950 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:24.806320 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:24.820679 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:24.826265 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:24.842674 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:24.848973 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:24.870445 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:24.902868 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:24.964185 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:24.986968 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:18:25.054900 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:25.162824 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:25.194615 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:25.248574 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:25.256412 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:25.271139 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:25.271405 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:25.282768 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:25.284847 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:25.320863 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:25.348558 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:25.402877 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:25.420372 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:25.438750 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:25.520479 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:25.522716 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:25.545066 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:25.555735 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:25.562865 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:25.563236 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:25.603226 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:25.603197 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:25.755625 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:25.794927 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:25.797724 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:25.806596 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:25.815171 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:25.836071 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:25.839654 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:25.842624 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:25.862458 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:25.909065 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:26.016454 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:18:26.023826 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:26.112967 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:18:26.182350 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:26.257364 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:26.260291 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:26.261175 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:26.264457 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:18:26.294042 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:26.296723 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:26.300784 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:26.385977 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:26.460749 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:26.492444 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:26.497134 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:18:26.530205 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:26.555538 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:26.555590 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:26.558226 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:26.604503 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:26.605357 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:26.617647 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:26.791719 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:26.797339 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:26.807471 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:26.812525 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 08:18:26.836308 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:26.838285 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:26.840745 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:26.843842 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:26.878322 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:26.921582 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:27.050040 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:27.110153 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:27.221208 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:27.221560 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:27.247727 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:27.283073 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:18:27.291115 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:27.294032 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:27.302201 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:27.308954 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:27.313730 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:27.325649 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:27.338372 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:27.434829 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:27.477731 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:18:27.557653 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:27.560050 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:27.560458 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:27.566453 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:27.569948 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:27.597491 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:27.602813 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:18:27.604456 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:27.675163 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:27.688787 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:27.690908 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:27.829083 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:27.829452 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:27.837433 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:27.841889 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:27.844213 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:18:27.854216 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:27.893563 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:27.899578 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:27.916770 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:27.933867 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:28.118074 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:28.153548 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:28.299760 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:28.335574 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:28.363245 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:18:28.405635 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:28.409945 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:28.417973 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:28.434335 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:28.434772 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:28.465310 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:28.496240 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:18:28.507783 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:28.693948 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:28.709703 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:28.710793 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:28.714701 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:28.728753 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:28.738713 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:28.744923 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:28.800427 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:28.832334 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:18:28.902747 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:28.915841 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:28.971900 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:28.973960 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:29.005942 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:29.007072 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:29.014200 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:29.023427 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:29.026947 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:29.055763 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:29.250233 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:29.270972 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:29.368049 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:29.416354 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:29.449913 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:29.469820 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:29.487974 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:29.503657 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:29.505905 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:29.513132 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:29.628338 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:29.698700 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:29.724298 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:29.740283 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:29.774626 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:29.789747 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:29.794636 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:29.805591 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:29.827831 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:29.828174 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:29.856299 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:29.862675 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:29.865471 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:29.865650 14423 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 52282.7us
W20240503 08:18:29.891578 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:30.051775 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:30.115731 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:30.126932 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:30.130597 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:30.168794 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:30.192895 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:30.211795 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:30.230091 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:30.252229 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:30.326483 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:30.328050 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:30.333921 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:30.339597 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:30.343796 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:30.349920 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:30.455832 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:30.595547 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:30.611562 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:30.628404 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:30.632542 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:30.633513 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:30.660612 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:30.662540 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:30.781790 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:30.787679 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:30.827414 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:18:30.880286 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:30.891824 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:30.904120 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:30.909224 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:30.912959 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:30.935971 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:30.959748 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:31.069675 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:31.098877 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:31.142832 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 119 similar messages]
I20240503 08:18:31.163555 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:31.173024 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:31.189723 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:31.191612 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:31.208645 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:31.214082 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:31.224042 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:31.260864 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:31.341699 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:31.343987 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:31.347842 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:31.393543 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:18:31.627012 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:31.629590 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:31.637104 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:31.640681 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:31.662185 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:31.662569 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:31.688905 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:31.716109 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:31.858105 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:31.889178 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:18:31.905436 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:31.913669 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:31.924968 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:31.930274 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:31.953016 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:31.958421 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:31.985348 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:32.104342 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:32.166930 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:32.173628 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:32.193106 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:32.198495 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:32.207003 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:32.222787 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:32.223346 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:32.250427 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:32.323045 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:32.328008 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:32.334929 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:32.403084 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:32.405025 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:18:32.406020 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:32.504369 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:32.529946 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:32.634650 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:32.646014 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:32.649713 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:32.658200 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:32.658476 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:32.674549 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:32.712625 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:32.826013 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:32.852568 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:32.852980 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:32.873809 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:32.876942 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:32.889333 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:32.896728 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:32.907394 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:32.911752 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:32.951874 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:32.955104 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:33.002049 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:33.098217 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:33.182502 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:33.194434 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:33.214583 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:33.214677 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:33.231132 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:33.241751 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:33.323383 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:33.400751 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:33.418972 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:33.421396 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:33.430122 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:33.445520 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:18:33.641197 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:33.656885 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:33.660643 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:33.688120 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:33.692638 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:33.703954 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:33.711783 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:33.723331 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:33.804785 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:33.897598 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:33.910291 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:33.927402 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:33.929152 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:33.988564 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:33.994412 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:33.997648 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:34.033831 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:34.034886 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:34.065415 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:18:34.067137 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:34.230791 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:34.240042 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:34.256314 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:34.256484 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:34.311915 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:34.317905 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:34.375541 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:34.381615 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:34.466578 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:34.562378 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:34.596469 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:34.647984 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:34.674057 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:34.710014 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:34.714340 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:34.722527 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:34.736215 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:34.747638 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:34.751629 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:34.768898 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:34.817633 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:34.909627 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:34.964100 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:34.974042 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:34.987057 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:34.989044 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:35.012840 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:35.023360 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:35.068173 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:35.165006 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:35.173552 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:35.223315 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:35.243824 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:35.244658 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:35.268932 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:35.270265 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:35.271239 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:35.272619 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:35.272945 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:35.301522 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:35.341001 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:35.396338 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:18:35.475360 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:35.680020 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:35.695129 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:35.701440 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:35.714545 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:35.715698 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:35.735404 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:35.747879 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:35.752864 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:35.753178 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:35.765242 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:35.816815 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:35.892343 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:18:35.896483 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:35.983305 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:35.993937 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:36.031368 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:36.049970 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:36.067579 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:36.067679 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:36.144098 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:18:36.149255 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:36.189200 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:36.258296 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:36.258466 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:36.259529 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:36.263594 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:36.285318 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:36.289841 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:36.309386 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:18:36.309428 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:36.344640 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:36.363902 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:36.414834 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:36.525372 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:36.586591 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:36.594404 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:36.739085 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:36.754920 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:36.755129 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:36.782220 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:36.791647 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:36.792156 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:36.888402 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:36.925334 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:18:36.949373 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:36.987327 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:36.989881 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:37.034865 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:37.038388 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:37.046981 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:37.056113 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:37.067935 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:37.104516 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:37.106078 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:37.149164 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:37.241586 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:37.274292 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:37.314796 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:37.324270 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:37.325817 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:37.349246 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:37.352113 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:37.379542 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:37.427589 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 08:18:37.484521 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:37.543205 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:37.561604 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:37.571650 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:37.574438 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:37.613845 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:37.631625 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:37.766062 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:37.785187 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:37.797799 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:37.803575 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:37.815711 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:37.818256 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:37.830761 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:37.965689 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:37.983740 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:38.048391 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:38.073467 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:38.086489 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:38.087284 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:38.103452 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:38.110901 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:38.117604 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:38.136744 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:38.141723 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:38.253551 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:38.354869 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:38.357841 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:38.365720 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:38.382624 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:38.387709 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:18:38.393891 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:38.416826 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:38.423749 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:38.459671 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:38.471244 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:38.564587 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:38.732285 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:38.786989 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:18:38.798766 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:18:38.823117 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:38.834084 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:38.841463 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:38.841475 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:38.853605 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:38.877562 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:38.878398 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:38.931950 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:39.014348 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:39.020582 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:39.081089 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:39.114100 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:39.128274 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:39.130496 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:39.145689 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:39.167251 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:39.174739 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:39.372175 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:39.372804 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:39.390741 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:39.392374 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:39.394027 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:39.408154 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:39.416848 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:39.422142 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:39.471534 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:39.531742 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:39.592499 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:39.642661 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:39.653429 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:39.802273 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:18:39.859069 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:39.870788 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:39.870999 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:39.890064 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:39.920094 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:39.920922 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:39.963132 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:18:39.984143 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:39.986294 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:39.996745 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:40.006057 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:40.158687 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:40.195801 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:18:40.199255 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:40.209725 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:40.210861 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:40.213342 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:40.232756 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:40.235234 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:40.320701 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:40.332484 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:40.361107 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:40.436190 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:40.458053 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:40.459208 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:40.462585 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:40.465186 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:40.491474 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:40.503978 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:18:40.510826 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:40.515040 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:40.564857 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:40.778867 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:18:40.779772 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:40.830710 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:40.847486 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:40.909904 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:18:40.949126 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:40.972396 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:40.981830 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:40.994130 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:41.002662 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:41.044142 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:41.064057 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:41.190891 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:18:41.191836 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:41.244606 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:41.282297 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:41.288162 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:41.295753 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:41.306291 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:41.314872 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:41.340574 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:41.350704 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:41.389343 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:41.436012 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:18:41.492466 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:41.506520 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:41.552270 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:41.560449 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:41.562451 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:41.582779 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:41.589834 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:41.605003 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:41.611086 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:41.731779 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:41.779274 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:41.893905 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:41.908730 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:41.930728 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:18:41.955454 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:42.039073 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:42.048600 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:42.065096 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:42.068344 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:42.085458 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:42.088827 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:42.102869 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:42.161957 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:42.206348 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:42.302403 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:42.349020 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:42.349979 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:42.357355 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:42.375779 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:42.377594 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:42.416311 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:42.420516 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:42.431413 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:42.554638 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:42.564096 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:42.602749 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:18:42.604972 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:18:42.615557 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:42.617067 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:42.625337 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:42.629696 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:42.660966 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:42.684180 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:42.686590 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:42.688074 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:42.805114 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:42.821380 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:42.849766 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:42.963832 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:43.094715 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:43.110451 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:43.129251 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:43.145624 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:43.147805 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:43.156258 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:43.175500 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:43.193171 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:43.195715 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:43.270484 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:43.274731 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:43.292443 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:43.379501 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:43.384181 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:43.387607 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:43.401368 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:18:43.409488 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:43.411124 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:43.415205 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:43.597975 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:43.645741 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:43.653112 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:43.664855 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:43.666476 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:43.700980 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:43.715029 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:43.719128 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:43.723542 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:43.737567 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:43.810735 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:18:43.812240 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:43.903824 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:44.035871 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:44.120086 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:44.126147 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:44.142481 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:44.145812 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:18:44.150378 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:44.160058 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:44.167411 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:44.173624 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:44.226285 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:44.378804 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:44.388751 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:44.398607 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:44.404383 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:44.417294 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:44.421245 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:44.473292 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:44.485486 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:44.558228 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:44.572757 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:44.576786 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:44.665820 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:44.677984 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:44.678658 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:44.679080 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:44.691365 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:44.693837 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:44.713810 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:44.821789 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:44.894848 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:44.926800 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.048945 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:18:45.150375 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:45.163403 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:45.163945 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:45.166093 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:45.171561 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:45.173388 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:45.189131 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:45.207891 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:45.250715 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:45.302343 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.349261 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.394968 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:45.395606 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:45.412600 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:45.412770 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:45.412995 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:18:45.420876 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:45.441210 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:45.443373 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:45.573998 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.644855 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:45.680305 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:45.698367 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:45.700996 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:45.718513 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:45.720232 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:45.722779 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:45.730417 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:45.809170 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.816980 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.927529 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.940825 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:45.953281 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:46.013121 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:46.030913 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:46.082830 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:46.150362 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:18:46.158141 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:46.170509 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:46.181393 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:46.188383 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:46.197170 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:46.224112 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240503 08:18:46.227717 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:46.228049 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:46.257948 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:18:46.345932 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:46.397594 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:46.409628 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:46.413606 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:46.425388 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:46.432407 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:46.438551 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:46.452301 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:46.518697 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:46.575789 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:46.599982 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:46.654490 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:46.668088 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:46.671360 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:46.671571 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:46.675453 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:46.694223 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:46.870111 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:46.894405 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:46.961797 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:47.051304 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:47.071132 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:47.102912 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:47.119810 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:47.138410 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:47.143903 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:47.153717 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:47.158545 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:47.168031 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:47.169629 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:47.265077 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:18:47.325435 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:47.331403 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:47.407518 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:47.427450 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:47.428781 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:47.440101 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:47.463781 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:47.477727 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:47.567095 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:47.576387 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:47.611759 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:18:47.655583 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:47.703711 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:47.717865 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:47.718237 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:47.731367 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:47.734107 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:47.743285 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:47.751535 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:47.772608 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:47.841588 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:48.134536 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:48.140813 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:48.164822 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:48.165906 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:48.175740 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:48.183948 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:48.184190 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:48.188783 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:48.192785 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:48.201933 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:48.204173 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:48.205905 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:48.344632 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:48.453452 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:48.453799 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:48.474381 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:48.477422 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:48.516381 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:48.567090 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:48.589325 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:48.593827 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:48.594581 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:48.649730 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:48.727409 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:48.731720 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:48.764827 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:48.771731 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:48.778103 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:48.779487 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:48.781164 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:48.789424 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:18:48.793460 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:18:48.798290 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:48.800797 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:48.821419 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:49.178638 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:49.211570 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:49.217010 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:49.227602 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:49.230484 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:49.232482 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:49.260643 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:49.275336 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:49.277329 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:49.287168 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:49.467265 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:49.475872 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:49.487483 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:49.507915 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:49.513775 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:49.524778 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:49.530375 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:49.553786 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:49.598641 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:49.707509 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:49.729852 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:49.752784 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:49.753520 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:49.777592 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:49.785929 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:49.803194 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:49.812275 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:49.881800 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:49.922866 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:49.971283 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:49.987870 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:49.993896 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:50.121320 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:50.226785 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:50.249020 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:50.255096 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:50.257980 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:50.279122 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:50.301697 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:50.311360 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:50.465816 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:50.493773 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:50.503614 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:50.505162 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:50.516156 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:50.539981 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:50.543757 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:50.577507 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:50.577903 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:50.584805 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:50.592252 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:50.608134 14685 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 61254.2us
I20240503 08:18:50.628688 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:50.639699 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:50.805083 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:50.815336 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:50.818444 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:50.830286 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:50.844900 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:50.850266 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:50.850307 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:50.924324 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:50.951296 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:50.974295 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:51.008709 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:51.037199 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:51.049007 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:51.161661 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:18:51.175495 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:18:51.176445 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:51.270090 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:51.276063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:51.297770 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:51.301968 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:51.305496 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:51.306815 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 116 similar messages]
I20240503 08:18:51.332185 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:51.352605 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:51.393110 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:51.408610 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:51.519543 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:51.533808 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:51.541376 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:51.555437 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:51.579790 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:51.590519 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:51.594748 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:51.695449 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:51.721776 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:51.758786 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:18:51.798516 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:51.801534 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:51.806818 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:51.811586 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:51.824834 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:51.847599 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:51.898176 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:51.928273 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:52.037494 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:52.120738 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:52.253460 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:52.253777 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:52.268637 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:52.278630 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:52.290774 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:52.304505 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:52.316485 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:52.332665 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:18:52.407047 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:52.432284 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:52.492172 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:18:52.496803 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:52.509841 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:52.512939 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:52.522671 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:52.522948 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:52.543848 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:52.559485 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:52.582723 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:52.599906 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:52.614483 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 08:18:52.737659 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:18:52.808205 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:52.828848 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:52.837260 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:52.843201 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:52.873322 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:52.883677 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:52.906747 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:52.940804 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:52.962281 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:53.008800 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:53.069970 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:53.070914 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:53.073354 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:53.282615 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:53.289561 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:53.289803 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:53.314443 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:53.315898 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:53.335490 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:53.392748 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:53.458858 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:53.461238 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:53.556383 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:53.558606 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:18:53.561811 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:53.566098 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:53.578938 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:53.587040 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:53.591933 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:53.594822 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:53.598413 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:53.605152 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:18:53.636822 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:53.789201 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:53.825750 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:53.840492 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:53.852478 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:53.853163 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:53.855487 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:53.855772 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:18:53.866500 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:53.885385 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:54.092273 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:54.147567 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:54.175634 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:54.273665 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:54.281205 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:54.282785 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:18:54.295740 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:54.302636 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:54.313747 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:54.315083 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:54.316322 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:54.349321 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:18:54.394376 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:54.545670 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:54.550770 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:54.558925 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:54.563897 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:54.579020 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:54.586571 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:54.635732 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:54.674537 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:54.704089 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:54.818509 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:54.818984 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:54.821877 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:54.829730 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:54.833082 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:54.848707 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:54.854029 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:54.912817 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:54.930856 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:54.991930 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:55.051529 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:55.178223 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:55.268383 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:55.278504 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:55.281148 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:55.288265 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:55.298427 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:55.316032 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:55.332091 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:55.403208 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:55.522325 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:55.524701 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:55.528518 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:18:55.532896 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:55.540139 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:55.543252 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:55.555337 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:55.582711 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:55.593422 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:55.605612 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:55.782619 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:55.784611 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:55.795147 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:55.809551 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:55.815620 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:55.818888 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:55.838907 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:55.858769 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:55.867549 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:55.893267 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:55.921259 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:55.921393 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:55.950911 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:56.019055 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:56.105227 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:56.180655 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:18:56.255761 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:56.268463 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:56.269511 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:56.276621 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:56.296460 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:56.321764 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:56.372148 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:18:56.373657 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:18:56.375170 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:56.378545 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:18:56.397897 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:56.460533 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:56.542511 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:56.553110 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:56.574687 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:56.603262 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:18:56.604563 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:56.620579 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:56.626036 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:56.630375 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:56.731891 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:56.803867 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:56.840834 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:56.851859 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:56.858379 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:56.858934 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:56.873912 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:56.879786 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:18:56.898101 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:56.927974 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:57.003548 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:57.052464 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:57.147255 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:18:57.284160 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:57.292858 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:57.300031 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:57.303797 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:57.337834 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:57.348382 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:57.376474 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:57.383103 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:57.431419 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:57.487079 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:18:57.515412 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:57.585449 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:57.592715 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:57.595063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:57.602286 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:57.632053 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:18:57.633864 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:57.650405 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:57.654412 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:57.692934 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:57.809837 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:57.838804 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 08:18:57.851476 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:57.873754 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:57.874361 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:57.902462 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:57.903453 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:57.926049 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:57.963999 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:58.011710 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:58.023995 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:18:58.147437 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:58.246919 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:58.282667 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:58.323361 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:58.323536 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:58.329375 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:58.330552 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:58.347496 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:58.349720 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:18:58.351469 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:58.370443 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:58.394943 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:18:58.504508 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:58.505034 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:58.557559 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:58.569907 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:18:58.600451 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:58.614486 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:58.645795 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:58.670778 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:58.674873 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:58.677309 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:58.683588 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:58.684363 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:58.825611 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:18:58.866353 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:58.867648 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:58.873304 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:58.893008 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:58.895324 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:58.896679 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:58.900076 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:18:58.923401 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:59.037285 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:18:59.073464 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:18:59.115121 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:18:59.115700 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:18:59.268263 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:59.326829 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:18:59.358412 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:59.358557 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:59.359308 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:59.360610 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:59.395224 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:18:59.396601 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:59.422895 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:18:59.588686 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:59.596625 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:59.597798 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:18:59.621626 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:18:59.650275 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:59.654525 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:59.659278 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:59.744639 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:59.778508 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:18:59.807991 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:18:59.840663 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:18:59.879235 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:18:59.881659 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:18:59.899024 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:18:59.901928 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:18:59.916663 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:18:59.945500 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:18:59.960819 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:18:59.978830 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:00.165860 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:00.217015 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:00.249044 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:00.281666 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:19:00.368567 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:00.375926 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:00.388613 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:00.392381 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:00.422025 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:19:00.450510 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:00.455077 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:00.594149 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:00.639103 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:00.649091 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:00.665141 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:19:00.666954 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:00.667953 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:00.672327 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:00.686772 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:00.706125 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:00.751154 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:00.875200 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:00.919749 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:00.936291 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:00.937539 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:00.939731 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:00.949029 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:19:00.955389 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:00.982919 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:00.985577 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:01.018816 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:01.027590 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:01.074621 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:01.090904 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:01.100416 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:01.203969 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:01.235219 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:01.289727 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:01.346729 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:01.406561 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:01.409480 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:01.413993 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:01.448267 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 123 similar messages]
I20240503 08:19:01.452279 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:01.455102 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:01.478998 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:01.567548 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:19:01.625681 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:19:01.626693 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:01.670598 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:01.683130 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:01.689270 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:01.720129 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:01.728731 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:01.736749 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:01.737543 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:01.749795 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:01.771147 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:01.948940 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:01.961968 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:01.963541 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:01.985816 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:01.997305 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:02.025053 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:02.060361 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:02.072974 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:02.139274 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:02.196835 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:02.214424 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:02.265542 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:02.372573 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:02.439788 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:02.446624 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:02.499253 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:02.506927 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:02.511155 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:02.518718 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:02.546756 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:02.555802 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:02.567020 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:02.670663 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:19:02.679641 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:02.713445 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:02.795500 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:02.799185 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:02.809986 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:02.828778 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:02.834110 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:02.835040 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:02.958009 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:03.028540 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:03.030952 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:19:03.034812 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:03.067385 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:03.083377 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:03.084141 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:03.088235 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:03.089128 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:03.096153 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:03.163806 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:03.164469 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:03.204169 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:03.350360 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:03.376822 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:03.459203 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:03.504076 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:03.505470 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:03.533282 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:03.547987 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:03.564671 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:03.570677 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:03.588579 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:03.628957 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:03.656373 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:03.679805 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:03.703421 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:03.746627 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:03.776190 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:03.786262 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:03.792358 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:03.796525 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:03.812350 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:03.820273 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:03.857626 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:03.867919 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:03.912832 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:03.932327 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:19:03.996889 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:19:04.024292 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:04.039588 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:04.053407 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:04.058629 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:04.078505 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:04.082012 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:04.088019 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:04.275895 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:04.334264 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:04.434391 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:04.477248 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:04.488945 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:04.497478 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:04.498189 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:04.502452 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:04.517057 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:04.519220 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:04.543318 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:04.626503 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:04.653889 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:04.655150 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:04.729400 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:04.740482 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:04.746771 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:04.751765 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:04.755471 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:04.773888 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:04.814875 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:04.895596 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:04.916277 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:04.998912 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:05.019238 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:05.042754 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:05.048310 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:05.101164 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:05.122793 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:05.137239 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:05.148219 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:05.175410 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:05.265286 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:05.474399 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:05.518632 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:05.544579 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:05.551443 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:05.552865 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:05.561328 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:05.576189 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:05.588191 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:05.590907 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:05.642647 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:05.664661 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:05.715924 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:05.760421 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:05.797538 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:05.807693 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:05.825738 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:05.829757 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:05.834406 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:05.873014 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:05.883117 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:06.005971 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:06.044566 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:06.127705 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:06.132989 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:06.148232 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:06.167901 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:06.172015 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:06.175853 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:06.183157 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:06.185482 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:06.185547 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:06.271250 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:06.298070 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:06.469063 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:06.469321 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:19:06.470130 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:06.529536 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:06.562002 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:19:06.595247 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:19:06.596493 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:06.600868 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:06.615175 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:06.630945 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:06.635537 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:06.644614 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:06.791740 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:06.793547 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:19:06.799213 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:06.835041 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:06.867149 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:06.883282 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:06.885696 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:06.890450 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:06.900727 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:06.916658 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:06.945091 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:06.966811 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:07.072721 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:07.135038 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:07.158187 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:07.175321 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:07.178777 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:07.181872 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:07.221699 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:07.238445 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:07.238617 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:07.279383 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:07.408002 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:07.627579 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:07.644801 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:07.657446 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:07.666913 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:07.669687 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:07.687448 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:07.706318 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:19:07.708451 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:07.709787 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:07.757616 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:07.763726 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:07.781229 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:07.894703 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:07.896003 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:07.925117 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:07.932101 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:07.933938 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:07.942744 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:07.953089 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:07.989260 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:08.042943 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:08.130810 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:19:08.148809 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:08.168148 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:08.178197 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:08.236889 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:08.245721 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:08.255396 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:08.263011 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:08.273774 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:08.324580 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:08.358595 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:08.401486 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:08.468102 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:08.708313 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:08.711705 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:08.738272 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:08.752708 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:08.763775 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:08.765507 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:08.770659 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:08.777079 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:08.790081 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:08.803320 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:08.821970 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:08.843750 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:08.897462 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:09.021353 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:09.030083 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:09.038192 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:09.076603 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:09.084873 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:09.094902 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:19:09.107594 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:09.199728 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:09.234036 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:09.298921 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:09.299206 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:09.300043 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:09.316002 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:09.326743 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:09.335728 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:09.352247 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:09.353070 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:09.360878 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:09.368844 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:09.398535 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:09.431510 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:19:09.736693 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:09.742341 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:09.753883 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:09.762248 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:09.777002 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:09.784809 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:09.815290 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:09.857087 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:09.859591 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:09.861871 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:09.913326 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:10.060381 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:10.087247 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:10.101996 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:10.109963 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:10.140105 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:10.176826 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:10.185582 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:10.198891 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:10.262715 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:10.343998 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:10.352355 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:10.358503 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:10.365453 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:10.365551 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:10.387039 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:10.397487 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:10.430958 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:10.480494 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:10.539243 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:10.610292 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:10.667390 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:10.756300 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:10.816408 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:10.833484 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:10.838672 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:10.848764 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:10.857847 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:10.878778 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:10.919478 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:10.936453 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:11.088676 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:11.088825 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:11.121311 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:19:11.133656 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:11.151432 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:11.175595 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:11.176479 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:11.189204 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:11.192998 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:11.203217 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:11.211014 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:11.226766 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:11.301323 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:11.397848 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:19:11.407908 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:11.424103 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:11.433862 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:11.434437 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:11.442715 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:11.464118 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:11.465175 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:11.490146 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:11.506251 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:19:11.575970 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:11.602982 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:11.710952 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:11.730363 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:11.754959 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:11.796155 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:11.883553 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:19:11.896021 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:11.902273 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:11.903895 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:11.916658 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:11.986619 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:12.022471 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:12.047770 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:12.115504 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:19:12.140740 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:12.206885 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:12.212225 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:12.214671 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:12.223944 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:12.227119 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:12.240257 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:12.243153 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:12.466763 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:12.476938 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:12.478843 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:12.499642 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:12.513345 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:12.538486 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:12.542089 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:12.553628 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:12.559839 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:12.561281 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:12.664230 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:12.726732 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:19:12.815593 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:12.833402 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:19:12.950706 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:12.953650 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:12.969844 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:12.985271 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:13.016379 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:13.033970 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:13.063830 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:13.125850 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:13.151679 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:13.156162 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:19:13.157323 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:13.164741 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:13.190449 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:19:13.221431 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:13.232367 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:13.240808 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:13.240900 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:13.252837 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:13.261932 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:13.269317 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:13.374848 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:13.399004 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:19:13.478431 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:13.481613 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:13.485512 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:13.495877 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:13.508170 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:13.514638 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:13.532649 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:13.573279 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:13.582949 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:13.605413 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:13.641572 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:13.806358 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:13.859349 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:19:14.036466 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:14.052690 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:14.053478 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:14.064230 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:19:14.067891 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:14.086856 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:14.089838 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:14.122738 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:14.132664 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:14.174705 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:14.219758 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:14.249514 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:14.257767 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:14.328534 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:14.329785 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:14.350127 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:14.350865 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:14.387109 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:14.391345 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:14.410248 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:14.570394 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:14.575336 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:14.582064 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:14.590461 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:14.607915 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:14.612002 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:14.645401 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:14.645910 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:14.716780 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:14.777861 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:19:14.948990 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:15.050462 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:15.064752 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:15.069043 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:15.088706 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:15.094599 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:15.118496 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:15.130553 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:15.150034 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:15.163990 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:15.214859 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:15.320880 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:15.330494 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:15.337062 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:15.345742 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:15.362946 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:15.383960 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:15.428481 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:15.539145 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:15.552940 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:15.558842 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:15.570262 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:15.579216 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:15.594029 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:15.594758 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:15.625089 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:15.627278 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:19:15.627888 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:15.651199 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:15.666939 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:15.722744 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:15.874336 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:15.895614 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:15.897397 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:15.928642 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:16.070582 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:16.071934 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:16.079023 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:16.079129 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:16.084502 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:16.092027 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:16.103993 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:16.129086 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:16.171257 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:16.173283 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:16.207412 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:16.331259 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:16.343055 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:16.348832 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:16.366958 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:16.368698 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:16.372603 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:16.400889 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:16.501101 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:19:16.529696 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:19:16.556361 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:16.564803 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:19:16.579859 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:16.590427 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:16.600965 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:16.619800 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:16.626397 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:16.655520 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:16.660451 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:16.679255 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:16.692451 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:16.798112 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:16.892341 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:16.998224 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:19:17.071532 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:17.081843 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:17.130869 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:17.132226 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:17.149502 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:17.161739 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:17.162250 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:17.173733 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:17.204979 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:17.312538 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:19:17.399247 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:17.399777 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:17.406107 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:17.407105 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:17.416441 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:17.419281 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:17.505698 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:17.529830 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:17.579326 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:17.661592 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:17.662460 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:17.675631 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:17.680613 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:17.685777 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:17.694445 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:17.702927 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:17.705639 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:17.728624 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:19:17.852849 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.039363 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:18.044381 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.064193 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:18.090489 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.106261 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.110122 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:18.130821 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:18.154779 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:18.175565 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:18.179203 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:18.189935 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:18.190735 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:18.197188 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:18.229290 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.249035 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.268250 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:19:18.448779 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:18.453220 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:18.453460 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:18.454878 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:18.461500 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:18.470546 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:18.477116 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.478554 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.499717 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:18.600975 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:18.628324 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:18.696377 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:18.714318 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:18.714566 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:18.719781 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:18.740263 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:18.771961 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:19:18.772840 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:18.799124 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:18.927853 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:19.015373 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:19.040171 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:19.064317 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:19.097537 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:19.195387 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:19.205212 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:19.209046 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:19.216616 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:19.233564 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:19.233971 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:19.251765 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:19.263341 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:19:19.263935 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:19.268241 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:19.450008 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:19.471690 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:19.476940 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:19.492470 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:19.496383 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:19.521776 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:19.524284 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:19.537546 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:19.587536 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:19.742614 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:19.749689 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:19.752604 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:19.785146 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:19.791751 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:19.795809 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:19.911780 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:19.942502 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:19.945641 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:19:19.985396 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:20.238677 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:20.254596 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:20.257344 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:20.274345 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:20.294560 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:20.295341 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:20.331475 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:20.362390 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:20.369058 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:20.370260 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:20.433471 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:20.501173 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:20.512374 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:20.523171 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:20.533283 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:20.546516 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:20.551016 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:20.562091 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:20.629613 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:20.652716 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:20.678222 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:20.710942 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:20.718648 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:20.749938 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:20.797525 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:20.806455 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:20.814373 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:20.815464 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:20.850912 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:20.854030 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:20.856665 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:19:20.857316 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:20.870328 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:20.949271 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:21.153231 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:21.232280 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:19:21.249255 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:21.296378 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:21.298935 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:21.299177 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:21.318395 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:21.318660 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:21.319761 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:21.328064 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:21.330775 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:21.340440 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:21.473261 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:21.533247 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 115 similar messages]
I20240503 08:19:21.557250 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:21.572595 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:21.575662 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:21.575927 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:21.578658 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:21.583987 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:19:21.606799 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:21.614988 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:21.768663 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:21.802709 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:21.822082 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:21.825464 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:21.826488 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:21.853181 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:21.872121 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:21.894577 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:21.907627 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:21.934902 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:21.948258 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:22.169536 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:22.178454 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:19:22.309746 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:22.320544 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:22.322396 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:22.329056 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:22.343744 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:22.344131 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:22.345388 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:22.352629 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:22.358686 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:22.494598 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:19:22.588476 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:22.602074 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:22.612910 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:22.643728 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:22.644213 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:22.650056 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:22.762368 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:19:22.774313 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:22.778061 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:22.812755 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:22.887102 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:22.887416 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:22.906647 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:22.907917 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:22.912943 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:22.921562 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:22.936074 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:22.948532 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:23.016408 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:23.048094 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:23.111606 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.161912 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.218802 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.244633 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.268916 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:23.278824 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.324530 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:19:23.325531 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:23.366035 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:23.366815 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:23.432523 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:23.455276 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:23.456815 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:23.464394 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:23.475065 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:23.476536 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:23.604501 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.685240 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:23.754034 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:23.767508 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:23.769448 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:23.790724 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:23.797091 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:23.799232 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:23.799583 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:23.799973 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:23.830557 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:23.863354 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:23.866891 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:24.046731 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:24.051710 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:24.051729 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:24.055402 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:24.064509 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:24.065449 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:24.084100 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:24.170045 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:19:24.344132 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:24.373147 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:24.396255 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:24.450524 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:24.487715 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:24.497911 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:24.502092 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:24.531961 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:24.534497 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:24.537583 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:24.630887 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:24.636221 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:24.745550 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:24.763531 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:24.770774 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:24.775527 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:24.782626 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:24.807309 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:24.813473 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:24.981122 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:25.015635 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:25.040858 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:25.047976 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:19:25.066288 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:25.084095 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:25.086645 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:25.094237 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:25.096518 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:25.223134 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:25.260314 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:25.335633 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:25.382678 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:25.392942 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:25.400422 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:25.522954 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:25.540688 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:25.547331 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:25.554435 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:25.572635 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:25.579931 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:25.615073 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:25.685407 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:25.690479 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:25.836001 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:25.855522 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:25.863400 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:25.868839 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:19:25.872771 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:25.880374 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:25.898480 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:25.903234 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:25.916838 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:25.919271 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:25.928280 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:25.948901 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:26.009990 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:26.165277 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:26.173918 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:26.174122 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:26.192133 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:26.194269 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:26.194736 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:26.244032 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:26.255163 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:26.272339 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:26.338014 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:26.478400 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:26.541682 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:26.551278 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:19:26.597925 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:26.648032 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:26.652902 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:26.653414 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:26.656041 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:26.678494 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:26.695354 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:26.732721 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:26.775884 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:19:26.893185 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:26.919978 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:26.920969 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:26.936421 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:26.953823 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:26.958446 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:26.995863 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:27.003492 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:27.082412 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:19:27.117777 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:27.124327 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:27.196986 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:27.200639 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:27.201467 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:27.205499 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:27.213860 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:27.227021 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:27.230973 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:27.234975 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:27.286860 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:19:27.334065 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:27.501963 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:27.613766 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:19:27.654769 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:27.654882 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:27.671108 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:27.674418 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:27.675693 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:27.677861 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:27.711669 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:27.754376 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:27.855142 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:19:27.857342 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:27.866389 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:19:27.926182 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:27.930735 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:27.943110 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:27.948348 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:27.958608 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:27.965463 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:27.970561 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:27.985400 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:27.999766 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:28.087896 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:28.151842 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:28.230278 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:28.246125 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:28.246381 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:28.256407 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:28.285679 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:28.286919 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:28.287987 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:28.389811 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:28.413612 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:28.457604 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:19:28.474778 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:28.499384 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:28.518759 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:28.546375 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:28.592608 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:28.729048 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:28.744966 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:28.750826 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:28.751596 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:28.764425 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:28.782253 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:28.918426 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:28.944018 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:29.045318 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:29.046375 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:29.046763 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:19:29.048255 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:29.056630 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:29.062776 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:29.064288 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:29.066260 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:29.069290 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:29.103924 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:29.118088 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:29.256687 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:29.323421 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:29.323942 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:29.339094 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:29.340415 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:29.356369 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:29.372607 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:29.468291 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:29.558255 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:29.595659 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:29.598204 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:29.607997 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:29.690932 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:29.786811 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:29.803843 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:29.804255 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:29.813283 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:29.822427 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:29.827522 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:29.831657 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:19:29.997150 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:30.014514 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:30.042608 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:19:30.058215 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:30.068559 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:30.074291 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:30.079936 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:30.080744 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:30.086822 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:30.099092 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:30.102505 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:30.336177 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:30.348500 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:30.355223 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:30.370332 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:30.394381 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:30.402118 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:30.413414 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:30.419770 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:30.454656 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:30.519697 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:30.534744 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:30.605760 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:30.620738 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:30.637166 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:30.807536 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:30.809597 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:30.827665 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:30.828956 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:30.843389 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:30.843724 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:30.879935 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:30.884693 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:30.962651 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:31.002857 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:31.026381 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:31.060231 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:31.081969 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:19:31.124264 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:31.133857 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:31.141294 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:31.148824 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:31.207785 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:31.208786 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:31.301903 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:31.392890 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:19:31.419351 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:31.424228 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:31.442097 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:31.445072 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:31.460750 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:31.463732 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:31.545818 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:31.572793 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:19:31.574421 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:31.611227 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:31.619510 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:31.690763 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:31.824421 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:31.882685 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:19:31.884752 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:31.891744 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:31.902906 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:31.920820 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:31.922032 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:31.938555 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:19:31.939919 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:32.049463 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:32.108897 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:32.179410 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:32.187201 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:32.189107 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:32.195789 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:32.195816 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:32.203859 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:32.204826 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:32.219800 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:32.225282 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:32.341462 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:19:32.364395 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:32.388530 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:32.436058 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:32.457468 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:32.458675 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:32.462807 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:32.474917 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:32.482144 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:32.528323 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:32.597466 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:19:32.615222 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:32.864548 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:32.889639 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:32.923310 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:32.946410 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:32.947772 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:32.949486 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:33.003504 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:33.015957 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:33.039881 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:33.051343 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:19:33.052441 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.077529 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:33.101833 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.105405 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:19:33.239792 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:33.259418 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:33.273222 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:33.273761 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:33.275292 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:33.283177 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:33.332587 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.399925 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:33.407455 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:33.486129 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:33.503341 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:33.512626 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:33.524922 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:33.545184 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:33.552381 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:19:33.554119 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:33.563493 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:33.575013 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.581386 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.610446 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:33.708555 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.806294 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:33.830993 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:33.939299 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:33.954355 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:33.958352 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:33.967675 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:33.990695 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:34.043891 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:34.063947 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:34.068148 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:34.125481 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:34.148557 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:34.261162 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:34.267694 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:34.271441 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:34.275069 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:34.278573 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:34.281066 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:34.301530 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:34.435518 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:34.496438 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:34.511093 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:34.511224 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:34.517037 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:34.531107 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:34.539347 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:34.569284 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:34.622443 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:34.646301 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:34.670231 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:34.685400 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:34.912359 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:34.918545 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:34.975848 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:19:34.983534 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:35.025933 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:35.039789 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:35.084586 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:35.088443 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:35.096231 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:35.112159 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:35.116961 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:35.139832 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:35.162813 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:35.205236 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:35.348008 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:35.348038 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:35.358434 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:35.359272 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:35.371328 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:35.389961 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:35.394557 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:35.412777 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:19:35.589057 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:35.597838 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:35.605656 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:35.607882 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:35.615053 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:35.625911 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:35.631973 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:35.687072 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:35.700548 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:35.855883 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:35.883635 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:35.965781 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:36.036597 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:36.065254 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:36.101303 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:36.105173 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:36.110352 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:36.112682 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:36.120030 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:36.140151 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:36.140355 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:36.196805 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:36.208698 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:36.243525 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:36.374779 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:36.375344 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:36.380524 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:36.392184 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:36.392658 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:36.401053 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:36.447598 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:36.462098 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:36.495646 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:36.588256 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 117 similar messages]
I20240503 08:19:36.609931 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:36.620664 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:36.639226 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:19:36.640882 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:36.643203 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:36.645963 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:36.652781 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:36.662473 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:36.717427 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:36.824759 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:36.837203 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:37.071943 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:37.079833 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:37.097847 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:37.103127 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:37.108096 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:37.111261 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:37.113159 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:37.153079 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:37.197067 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:37.226182 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:37.237023 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:19:37.345610 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:37.353206 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:37.364331 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:37.384959 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:37.425233 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:37.426291 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:37.429972 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:37.448328 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:37.450470 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:37.455551 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:37.565505 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:19:37.628852 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:19:37.659310 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:37.666599 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:37.684878 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:37.689692 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:37.691854 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:37.692020 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:37.699332 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:37.897711 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:37.924935 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:37.943238 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:37.954142 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:38.021353 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:38.090363 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:38.113600 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:19:38.124908 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:38.138119 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:38.148700 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:38.154121 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:38.156272 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:38.170522 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:38.182308 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:38.261091 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:38.308377 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:19:38.383139 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:38.397583 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:38.415205 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:38.417328 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:38.446779 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:19:38.448740 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:38.488507 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:38.559692 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:38.579295 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:19:38.641353 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:38.654580 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:38.656576 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:38.700961 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:38.705873 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:38.720404 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:38.723110 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:19:38.755461 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:38.776253 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:38.791972 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:38.817082 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:39.000630 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:39.052982 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:39.103140 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:39.143620 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:39.149833 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:39.175523 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:39.200244 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:39.205376 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:39.216840 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:39.226351 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:39.229246 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:39.235459 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:39.345983 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:39.443877 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:39.453404 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:39.458915 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:39.466713 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:39.479743 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:39.480950 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:39.495218 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:39.572472 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:39.591059 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:39.592515 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:39.621543 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:39.714944 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:39.719043 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:39.719017 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:39.746876 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:39.746907 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:39.751231 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:39.770973 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:39.777088 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:40.076419 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:40.129452 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:40.178066 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:40.188880 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:40.199925 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:40.205534 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:40.206955 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:40.208076 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:40.210456 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:40.215629 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:40.306046 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:40.335587 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:19:40.371136 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:40.460659 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:40.477325 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:40.484359 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:40.510718 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:40.529132 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:40.529179 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:40.531329 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:40.569799 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:40.577787 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:40.706179 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:40.763785 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:40.782722 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:40.789264 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:40.791025 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:40.796324 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:40.814925 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:40.820117 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:40.875221 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:40.929237 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:41.072638 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:41.202597 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:41.219583 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:41.239140 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:19:41.239706 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:41.240999 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:41.254489 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:41.257750 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:41.261772 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:41.283970 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:41.300555 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:41.388118 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:41.431766 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:41.462317 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:41.465546 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:41.473789 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:41.480830 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:41.497607 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:41.509152 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:41.515980 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:41.516166 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:41.551192 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:41.569491 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:41.641062 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:19:41.673308 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:41.747496 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:41.750015 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:41.761745 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:41.784375 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:41.785135 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:41.813288 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:41.819048 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:41.904991 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:41.909632 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:41.915942 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:41.975678 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:42.098805 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:42.166213 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:42.167531 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:42.229367 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:42.250587 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:42.251682 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:42.266567 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:42.289245 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:42.294443 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:42.312238 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:42.466001 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:42.489346 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:42.490633 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:42.490798 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:42.527568 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:42.531463 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:42.586892 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:42.630790 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:19:42.640064 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:42.736063 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:19:42.738274 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:42.749668 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:42.758203 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:42.775595 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:42.793937 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:42.796437 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:42.814584 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:42.817255 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:42.833953 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:42.838773 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:42.890338 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:42.981384 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:43.073031 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:43.103204 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:43.125736 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:19:43.248085 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:43.251982 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:43.253657 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:43.256084 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:43.279718 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:43.284657 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:43.308513 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:43.325403 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:43.444455 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:43.492856 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:43.508972 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:43.524149 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:43.526413 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:43.528151 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:43.543047 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:43.572988 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:43.580557 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:43.676138 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:43.683732 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:43.781940 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:43.798745 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:19:43.799700 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:19:43.800603 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:43.801030 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:43.814824 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:43.829751 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:43.844230 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:43.870939 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:43.889988 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:43.909680 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:43.953632 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:44.019806 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:44.118968 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:44.224365 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:44.234300 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:44.250211 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:44.255429 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:44.258792 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:44.273053 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:44.274410 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:44.276875 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:44.281317 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:44.369671 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:44.393343 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:44.406049 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:44.437625 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:19:44.475102 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:19:44.490675 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:44.501889 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:44.513289 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:44.523888 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:44.556869 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:44.590400 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:44.646600 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:44.692121 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:44.780774 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:44.809213 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:44.810539 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:44.811606 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:44.843955 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:44.855325 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:44.883162 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:45.146766 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:45.263789 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:45.267539 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:45.286350 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:19:45.301487 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:45.304399 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:45.311995 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:45.323899 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:19:45.339380 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:45.339522 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:45.355108 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:45.382552 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:45.402859 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:45.494789 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:19:45.563482 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:45.575501 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:45.576642 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:45.579079 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:45.598345 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:45.620210 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:19:45.621420 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:45.804174 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:45.830200 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:45.839313 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:45.845397 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:45.848116 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:45.865710 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:45.892375 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:45.902354 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:45.916464 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:45.917586 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:45.971189 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:45.971438 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:46.103734 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:46.137766 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:46.265885 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:46.325119 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:19:46.329265 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:46.331808 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:46.351166 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:19:46.355751 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:46.356289 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:46.362597 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:46.370149 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:46.375164 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:46.459715 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:46.538412 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:46.623720 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:46.630304 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:46.644265 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:46.649924 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:46.653566 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:19:46.682271 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:19:46.687407 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:46.687928 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:46.777487 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:46.845703 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:46.846300 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:46.876652 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:46.888509 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:46.896163 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:46.901290 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:46.916103 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:46.961330 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:46.975198 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:46.982069 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:47.012635 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:47.102826 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:47.143150 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:47.350318 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:47.374645 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:47.374878 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:47.385762 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:47.402642 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:47.406708 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:47.420738 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:47.439076 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:47.464406 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:47.505509 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:47.606748 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:47.650066 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:47.675276 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:47.677115 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:47.702271 14814 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 45652.1us
I20240503 08:19:47.703651 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:47.703676 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:47.723528 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:47.828094 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:19:47.828987 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:47.867449 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:47.919960 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:47.943815 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:47.944070 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:47.944273 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:47.969650 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:19:47.974310 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:48.007881 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:48.037716 14686 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.136:37641, user_credentials={real_user=slave}} blocked reactor thread for 77809us
I20240503 08:19:48.038116 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:48.064316 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:48.088507 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:48.202904 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:19:48.203874 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:48.362607 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:48.371516 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:48.385341 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:48.402453 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:48.417475 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:48.460589 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:48.476321 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:19:48.496430 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:48.501521 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:48.514397 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:48.526006 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:48.528172 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:48.545598 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:48.597790 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:48.701318 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:48.735037 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:19:48.755422 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:48.761104 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:48.791426 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:48.811424 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:48.826859 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:48.832723 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:48.891909 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:19:48.958950 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:48.969118 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:48.989176 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:49.084486 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:49.095449 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:49.113448 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:49.113444 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:49.124082 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:49.132344 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:49.151892 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:49.172294 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:49.212440 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:49.317636 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:49.359961 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:49.511492 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:49.536557 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:49.537989 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:49.545977 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:49.548609 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:49.559568 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:49.573138 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:49.598524 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:49.616078 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:49.682662 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:49.765769 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:49.818029 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:49.818200 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:49.826838 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:49.829700 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:49.836136 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:49.849782 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:49.874004 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:49.926353 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:49.982628 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:50.073786 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:50.087862 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:50.089197 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:50.099344 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:50.115476 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:50.124997 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:50.151057 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:50.264773 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:50.385095 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:19:50.393379 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:50.395242 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:50.411806 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:50.509534 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:50.548080 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:50.561278 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:50.569442 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:50.575021 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:50.586493 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:50.600553 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:50.616626 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:50.819478 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:50.821306 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:50.823711 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:50.830061 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:50.834275 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:50.835034 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:50.854578 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:50.871281 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:50.910128 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:50.921267 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:50.966799 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:51.009199 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:51.075557 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:51.087231 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:51.094013 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:51.097302 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:51.106747 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:51.114488 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:51.121628 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:19:51.124572 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:51.138594 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:51.271225 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:51.412768 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:51.444339 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:51.471813 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:51.535429 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:51.556602 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:51.577258 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:51.581887 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:51.582427 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:51.614711 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:51.652637 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:51.692343 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:51.692610 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:19:51.702841 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:51.801993 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:51.832736 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:51.863286 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:51.882294 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:51.900009 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:51.910466 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:51.913954 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:51.956676 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:51.960716 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:52.063262 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:52.063980 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:52.104139 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:52.218858 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:52.225898 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:52.230527 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:52.252763 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:19:52.254091 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:52.264835 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:52.267333 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:52.270417 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:52.393548 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:52.495718 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:52.653067 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:52.667804 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:52.673924 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:52.674770 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:52.678337 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:52.706064 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:52.736074 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:52.737427 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:52.744297 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:52.853873 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:19:52.897991 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:52.912669 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:52.944089 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:52.957135 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:52.972174 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:52.980106 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:53.033895 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:53.055814 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:53.066871 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:19:53.067854 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:53.157313 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:53.195770 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:53.228446 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:53.240278 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 08:19:53.241115 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:53.241235 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:53.248708 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:53.255015 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:53.274798 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:53.297717 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:53.528403 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:53.534031 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:53.542444 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:53.563766 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:53.579769 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:53.595124 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:53.697376 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:53.701138 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:53.702252 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:53.709945 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:53.730878 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:53.733054 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:53.825515 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:53.835614 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:53.887393 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:53.894299 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:53.998032 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:54.025508 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:19:54.052974 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:54.068882 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:54.069397 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:54.110209 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:54.112339 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:54.125917 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:54.126791 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:54.129881 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:54.140381 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:54.205765 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:54.370993 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:54.376058 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:54.382371 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:54.396778 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:54.396978 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:54.416836 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:54.419986 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:54.446204 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:54.456640 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:54.465060 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:54.657101 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:54.684469 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:54.762197 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:54.816826 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:54.818187 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:19:54.820676 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:54.842141 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:54.861104 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:54.885394 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:54.886408 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:54.955090 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:55.041021 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:55.123946 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:55.127723 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:55.140223 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:55.141055 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:55.149199 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:55.156877 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:55.170536 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:55.173143 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:55.224017 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:55.232290 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:19:55.364717 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:55.387825 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:55.392218 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:55.394733 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:55.402401 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:55.405293 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:55.414316 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:55.475211 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:19:55.546358 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:55.559149 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:55.629009 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:55.722493 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:55.834007 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:55.837828 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:55.844085 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:55.846590 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:55.859622 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:55.861279 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:55.862306 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:55.957695 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:55.987243 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:56.060794 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:56.071326 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:56.132212 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:19:56.152524 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:56.166114 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:56.167989 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:56.180783 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:56.183991 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:56.189100 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:56.287724 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:56.355297 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:56.355911 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:56.400427 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:56.404186 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:56.408802 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:56.416123 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:56.420833 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:56.421451 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:56.427888 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:56.561311 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:56.647528 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:56.650097 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:56.737105 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:19:56.756059 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:56.770984 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:56.799304 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:56.804896 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:56.835256 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:19:56.846598 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:56.860641 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:56.862579 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:56.881608 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:56.896984 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:56.956322 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:57.103850 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:57.141062 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:57.154414 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:57.171298 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:57.179850 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:57.180749 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:57.189769 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:57.227532 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:57.247104 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:57.251260 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:19:57.422200 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:57.427345 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:57.434882 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:57.435545 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:57.453104 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:57.462481 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:57.482535 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:19:57.617506 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:57.632794 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:57.640661 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:57.665941 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:57.710238 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:19:57.737646 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:57.768615 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:19:57.868003 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:57.878774 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:57.881902 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:57.893674 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:57.893895 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:57.901953 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:57.911334 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:57.976564 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:19:58.151089 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:58.154109 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:58.172117 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:58.173442 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:58.175784 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:58.182674 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:58.220522 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:58.231890 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:19:58.243968 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:58.303287 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:58.314298 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 08:19:58.405112 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:19:58.407727 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:19:58.446882 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:58.448411 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:58.448980 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:58.472707 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:58.475342 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:58.475929 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:58.580078 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:58.583441 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:58.711441 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:58.742687 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:58.787703 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:58.827540 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:58.839799 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:19:58.906378 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:58.920387 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:58.937834 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:58.971117 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:59.023327 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:59.028025 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:19:59.135295 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:59.138630 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:59.160472 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:19:59.161432 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:19:59.176606 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:19:59.215216 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:59.227180 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:59.230063 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:59.230425 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:59.242497 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:59.242978 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:59.267187 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:19:59.365185 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:19:59.392771 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:19:59.401594 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:59.434868 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:19:59.465313 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:59.477516 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:59.482547 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:59.491827 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:19:59.513208 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:19:59.522981 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:19:59.597611 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:19:59.658428 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:19:59.703066 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:19:59.784291 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:19:59.934427 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:19:59.952106 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:19:59.958636 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:19:59.965420 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:19:59.993893 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:19:59.999056 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:00.049063 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:00.172372 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:00.182786 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:20:00.212853 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:00.246919 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:00.248826 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:00.259073 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:00.269063 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:00.279742 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:00.280160 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:00.294895 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:00.304711 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:00.322506 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:20:00.363519 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:00.450641 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:20:00.510733 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:00.518532 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:00.523769 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:20:00.531976 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:00.546979 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:00.602255 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:00.622746 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:00.651232 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:00.693985 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:00.756307 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:00.845780 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:00.881451 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:01.036306 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:20:01.051460 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:01.056747 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:01.058308 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:01.072835 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:01.080925 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:01.084707 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:01.114496 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:01.209947 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:01.268563 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:01.281914 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:01.289753 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:01.318924 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:01.319705 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:01.329607 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:01.330534 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:01.360622 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:01.368889 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:01.392354 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:01.528798 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:20:01.569895 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:01.582259 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:01.590818 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:01.602751 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:01.605893 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:01.613324 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:01.619489 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:01.728278 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:01.732717 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:01.797255 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 08:20:01.798272 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:01.875697 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:01.922267 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:01.960074 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:01.965956 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:01.982151 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:20:02.059933 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:02.074749 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:02.081730 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:02.088269 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:02.116688 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:02.171382 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:02.203984 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:02.226833 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:02.332520 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:02.346021 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:20:02.349030 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:02.357334 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:02.358723 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:02.377028 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:02.391569 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:02.395892 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:02.603143 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:02.612457 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:02.626775 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:02.641002 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:02.644474 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:02.673763 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:02.680301 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:02.697233 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:20:02.698673 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:02.714684 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:02.866235 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:02.876881 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:20:02.932248 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:02.993186 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:03.103844 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:03.118324 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:03.124858 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:03.128762 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:03.142367 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:03.158525 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:03.188058 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:03.260535 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:20:03.261493 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:03.275339 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:03.316769 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:20:03.345068 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:03.397737 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:03.402251 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:03.423404 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:03.433142 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:03.440279 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:03.452077 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:03.475915 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:03.492075 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:03.517313 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:20:03.665297 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:03.682099 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:03.692782 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:03.694483 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:03.706076 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:03.730191 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:03.734800 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:03.835474 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:03.844251 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:03.847101 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:03.855163 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:03.886672 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:04.004199 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:04.027562 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:04.133667 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:04.139904 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:04.157127 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:04.171885 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:04.196921 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:04.205170 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:04.246248 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:04.284559 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:04.318904 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:20:04.319690 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:04.357749 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:04.380383 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:04.394894 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:04.405257 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:04.405980 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:04.406911 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:04.428766 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:04.439589 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:04.457362 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:04.457510 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:04.457702 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:04.579011 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:04.641641 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:04.643810 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:04.655565 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:04.656462 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:04.671736 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:04.680387 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:04.824223 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:04.830857 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:20:04.909060 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:04.942222 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:05.099892 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:05.107493 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:05.109709 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:05.122503 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:05.129695 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:05.176369 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:05.333112 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:05.339274 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:05.347427 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:05.359515 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:05.365676 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:05.375363 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:05.384043 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:05.389137 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:05.394587 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:05.396399 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:05.473652 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:05.512774 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:05.609054 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:05.620155 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:20:05.631387 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:05.633924 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:05.654884 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:05.662744 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:05.672695 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:05.682898 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:05.867308 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:05.869418 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:05.973955 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:06.003162 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:06.085978 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:06.095309 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:06.106379 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:06.116442 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:06.155176 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:06.167541 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:06.170397 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:06.193459 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:06.195060 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:06.296124 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:06.337827 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:06.360723 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:06.384467 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:06.395430 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:06.397054 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:06.412672 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:06.415287 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:06.420049 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:06.430291 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:06.445299 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:06.481227 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:06.600577 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:06.610607 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:06.669658 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:06.671978 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:06.671978 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:06.677479 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:06.684021 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:06.704458 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:06.796344 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:06.842008 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:20:06.842939 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:06.848731 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:06.853551 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:06.899145 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:07.105865 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:07.123068 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:07.126914 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:07.132812 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:20:07.135546 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:07.146879 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:07.168104 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:07.172354 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:07.186872 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:07.245885 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:07.420329 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:07.432989 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:07.439342 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:07.441524 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:07.444218 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:07.450191 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:07.461736 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:07.524966 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:07.640180 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:07.703791 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:07.706357 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:07.713346 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:07.715659 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:07.724078 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:07.728413 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:07.917157 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:07.994014 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:08.021641 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:08.060160 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:20:08.096496 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:08.120473 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:08.154951 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:08.157490 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:08.167229 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:08.190826 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:08.210386 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:08.216573 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:08.259101 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:08.308785 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:20:08.309688 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:08.311719 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:08.397877 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:08.404345 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:08.411952 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:08.425340 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:08.437283 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:08.438656 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:08.446586 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:20:08.474653 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:08.600978 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:08.617103 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:08.648412 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:08.657424 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:08.666035 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:08.670617 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:08.688645 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:08.690429 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:08.695192 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:08.711879 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:08.723407 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:08.805193 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:08.837324 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:08.894387 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:08.899739 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:08.988034 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:09.135840 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:09.145547 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:09.148497 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:09.152441 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:09.160235 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:09.184571 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:09.208575 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:09.210089 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:09.230122 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:09.241734 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:09.250173 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:09.388442 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:09.401367 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:09.428783 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:09.429481 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:09.442435 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:09.456507 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:09.461484 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:09.507350 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:20:09.548615 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:09.607784 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:09.627084 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:20:09.684995 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:09.690747 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:09.700562 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:09.716717 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:09.732568 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:20:09.739699 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:09.757062 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:09.761346 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:09.824427 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:10.011413 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:10.158128 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:10.162671 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:10.163760 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:10.164517 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:10.168398 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:10.196290 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:10.256125 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:10.366094 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:10.385651 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:10.388796 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:10.402276 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:10.406250 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:10.418429 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:10.421804 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:10.423818 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:10.434388 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:20:10.470849 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:10.607992 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:10.633582 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:10.658685 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:10.660866 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:10.671236 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:10.685127 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:10.714982 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:10.727437 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:10.768204 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:10.869442 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:10.874940 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:10.886186 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:10.911487 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:11.078226 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:11.117319 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:11.120105 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:11.120992 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:11.139271 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:11.140525 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:11.165560 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:11.167263 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:11.193946 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:11.300372 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:11.301339 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:11.306728 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:11.327927 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:11.354483 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:11.368381 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:11.377254 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:11.396214 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:20:11.405304 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:11.422349 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:11.447635 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:11.465503 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:11.465657 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:11.486115 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:11.663134 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:11.664343 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:11.665695 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:11.674482 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:11.678290 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:11.694623 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:11.702811 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:11.709697 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:11.796010 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:11.942046 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:11.943980 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:20:11.961944 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:11.963189 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:12.000370 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:12.050019 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:12.069710 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:12.120005 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:12.138303 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:12.138556 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:12.156489 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:12.175953 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:12.183459 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:12.233382 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:12.340184 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:12.347735 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:12.409619 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:12.416774 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:12.417836 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:12.429596 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:12.436571 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:12.451695 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:12.475013 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:12.673417 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:12.686302 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:12.691290 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:12.700284 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:12.713728 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:12.718360 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:12.738425 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:12.914065 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:12.957633 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:20:12.986356 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:13.048535 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:13.070814 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:13.105451 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:13.123385 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:13.132385 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:13.134330 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:13.150913 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:13.154577 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:13.169114 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:13.178884 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:13.361608 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:13.378999 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:13.382416 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:13.383734 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:13.401667 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:13.402177 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:13.423102 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:13.518931 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:13.519107 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:13.529785 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:20:13.576709 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:20:13.577873 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:13.600458 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:13.631064 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:13.647038 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:13.651914 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:13.654045 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:13.668805 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:13.680611 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:13.684988 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:13.722458 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:13.753583 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:13.783092 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:13.860019 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:13.897987 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:13.905468 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:13.944805 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:14.058640 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:14.058854 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:14.074708 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:14.086830 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:14.107530 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:14.114641 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:14.132210 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:14.137828 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:14.150279 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:14.195245 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:14.198441 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:20:14.324725 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:14.326321 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:14.329012 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:14.330608 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:14.347594 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:14.357787 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:14.361768 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:14.379710 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:14.379997 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:14.452880 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:14.496635 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:14.560361 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 08:20:14.597163 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:14.610267 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:14.619868 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:14.624514 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:14.638646 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:14.682411 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:14.805907 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:14.979730 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:15.000946 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:15.059046 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:20:15.091782 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:15.096354 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:15.101578 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:15.107722 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:15.176630 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:15.194074 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:15.210425 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:15.307540 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:15.378959 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:15.383389 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:15.392248 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:15.401978 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:15.403928 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:15.407732 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:15.450910 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:20:15.461486 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:15.471124 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:15.493288 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:15.599781 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:15.655023 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:15.669253 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:15.675521 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:15.682502 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:15.712109 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:15.717365 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:15.771719 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:15.874833 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:15.962054 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.046259 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:20:16.108808 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:16.117173 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:16.119695 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:20:16.131759 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:16.137141 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:16.138383 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:16.163409 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:16.175220 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.194257 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:16.207739 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.222088 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.237272 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:16.256949 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.291196 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:20:16.365674 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:16.366245 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:16.376248 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:16.378367 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:16.404901 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:16.406589 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:16.427611 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:16.505579 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:16.552996 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.618862 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:20:16.643278 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:16.647753 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:16.652375 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:16.654677 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:16.656283 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:16.667843 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:16.721096 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:20:16.755347 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:16.765168 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:16.832228 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:16.876888 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:16.923977 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:16.951804 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 08:20:16.969841 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:20:17.114871 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:17.135113 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:17.135813 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:17.136282 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:17.187819 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:17.211627 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:17.228852 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:17.370609 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:17.373262 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:17.381129 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:17.389999 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:17.391340 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:17.399251 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:17.415328 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:17.419939 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:17.614974 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:20:17.637815 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:17.642903 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:17.645779 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:17.661926 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:17.667219 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:17.678483 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:17.709772 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:17.849817 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:17.879261 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:17.917104 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:18.010526 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:18.049747 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:20:18.116221 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:18.133992 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:18.153872 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:18.162523 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:18.191030 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:18.192075 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:18.250810 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:18.336611 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:18.384581 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:18.398823 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:18.407230 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:18.418756 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:18.421878 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:18.433403 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:18.444499 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:18.470645 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:18.642583 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:18.645728 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:18.646009 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:18.661900 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:18.666066 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:20:18.667603 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:18.670996 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:20:18.695149 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:18.696362 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:18.707252 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:18.709709 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:18.711046 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:18.721542 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:18.746778 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:18.748934 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:18.763847 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:18.783412 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:18.836990 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:18.850657 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:18.879660 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:18.998963 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:19.107602 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:19.111516 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:19.129280 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:19.131203 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:19.132257 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:19.140561 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:19.163615 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:19.230963 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:19.252509 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:19.336210 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:19.374852 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:19.403002 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:19.408818 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:19.409766 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:19.409857 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:19.418401 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:20:19.437229 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:19.437994 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:19.440168 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:19.443037 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:20:19.662667 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:19.675968 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:19.686254 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:19.687168 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:19.688294 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:19.718897 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:19.721704 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:20:19.777583 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:19.843624 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:20.037189 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:20.066287 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:20:20.112382 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:20.121891 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:20.152547 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:20.153723 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:20.161391 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:20.163215 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:20.187288 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:20.246095 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:20:20.281180 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:20.308063 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:20:20.359400 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:20.366719 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:20.379736 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:20.381803 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:20.394888 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:20.408736 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:20.494693 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:20:20.612569 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:20.630533 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:20.631457 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:20.666677 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:20.678452 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:20.685209 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:20.690364 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:20.698205 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:20.711263 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:20.718413 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:21.084167 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:21.096438 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:21.097770 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:21.100765 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:21.121572 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:21.132612 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:21.148273 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:21.150430 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:21.166476 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:20:21.168067 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:21.193709 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:21.261345 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:21.261737 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:21.277724 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:21.288249 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:21.318697 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:21.365499 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:21.366350 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:21.408535 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:21.443432 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:21.450974 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:21.458266 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:20:21.474359 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:21.492727 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:21.519403 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:21.535130 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:21.600732 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:21.637970 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:21.644188 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:21.719815 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:21.735167 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:21.745378 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:21.769090 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:21.775111 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:21.776392 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:21.833530 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:21.872377 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:21.914327 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:21.951937 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:20:21.952847 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:22.032402 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:22.199194 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:22.214084 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:22.222553 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:22.247198 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:22.255858 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:22.268594 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:22.282133 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:22.419301 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:22.475126 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:22.481681 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:22.495144 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:22.497002 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:22.498399 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:22.502070 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:22.502810 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:22.519716 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:22.687474 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:22.713868 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:22.721534 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:22.735409 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:22.754527 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:22.774675 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:22.801308 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:22.803411 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:22.823143 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:22.960121 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:23.042704 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:23.134950 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:23.182915 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:23.201609 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:23.208034 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:23.212447 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:23.228358 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:23.233219 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:23.252990 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:23.264554 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:20:23.380132 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:23.459182 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:23.464370 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:23.465276 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:23.469375 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:23.486464 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:23.505728 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:23.511848 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:23.610996 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:23.653328 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:23.681097 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:20:23.681982 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:23.694497 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:23.701756 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:23.751770 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:23.761538 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:23.774230 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:23.775974 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:23.793715 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:23.794742 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:23.796155 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:23.800678 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:23.807680 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:20:23.822057 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:23.825456 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:23.853168 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:23.863427 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:23.968178 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:24.043553 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:24.183455 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:24.208874 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:24.226763 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:24.257527 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:24.270622 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:24.272737 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:24.277101 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:24.281159 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:24.282457 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:24.287892 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:24.349784 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:24.499050 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:24.509016 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:24.516470 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:24.524458 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:24.538753 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:24.547169 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:24.614892 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:20:24.623117 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:24.646986 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:24.787420 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:24.803400 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:24.807199 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:24.810945 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 08:20:24.826445 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:24.855440 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:24.861864 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:24.867906 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:24.883590 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:25.082597 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:25.207981 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:25.233731 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:25.274147 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:25.283603 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:25.287446 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:25.289971 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:25.303045 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:25.321035 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:25.327227 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:25.388679 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:25.465629 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:25.557410 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:25.568578 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:25.599388 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:25.600469 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:25.604698 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:25.607969 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:25.663236 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:25.773816 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:25.775529 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:25.820225 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:20:25.821610 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:25.823136 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:25.830274 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:25.830916 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:25.838647 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:25.843452 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:25.877995 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:25.955224 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:26.038834 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:26.099491 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:26.113626 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:26.118628 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:26.206856 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:26.281097 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:26.310860 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:26.320214 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:26.327607 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:26.333287 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:26.336045 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:26.356722 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:26.358282 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:20:26.362079 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:26.369927 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:26.374967 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:26.388458 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:26.406944 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:26.517657 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:26.520538 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:26.607926 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:26.632561 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:26.645861 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:26.651651 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:26.666213 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:26.670148 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:26.722307 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:26.773960 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:26.849088 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:26.877858 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:26.886765 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:26.894631 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:26.920248 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:26.945686 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:26.946699 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:26.948652 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:27.032728 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:27.046626 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:20:27.212677 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:27.219007 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:27.220000 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:27.373667 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:27.388904 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:27.393815 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:27.394984 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:27.423673 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:27.424472 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:27.463814 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:27.503675 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:27.667783 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:27.673280 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:27.683349 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:27.688185 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:27.702273 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:27.703066 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:27.708959 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:27.718991 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:27.768322 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:27.926048 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:27.942801 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:27.954636 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:27.985206 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:28.000239 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:28.035537 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:28.067802 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:28.089814 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:28.125937 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:28.130806 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:20:28.198128 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:28.262207 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:28.271900 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:28.398391 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:28.534808 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:28.540622 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:28.561915 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:28.564386 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:28.575855 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:28.592492 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:28.599072 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:28.622116 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:28.698196 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:20:28.699168 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:28.700088 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:28.709091 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:28.712615 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:28.713469 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:28.722448 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:28.731727 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:28.736141 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:28.817822 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:28.822887 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:28.824545 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:28.826455 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:28.829157 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:28.835448 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:28.869032 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:28.885344 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:29.004550 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:20:29.089337 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:29.097623 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:29.100579 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:29.114046 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:29.118512 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:29.122632 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:29.136027 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:29.161151 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:29.197947 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:29.227632 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:29.360801 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:29.537920 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:29.558223 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:29.567061 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:29.572782 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:29.573066 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:29.580729 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:29.581811 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:29.584276 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:29.607867 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:29.623771 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:29.749290 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:29.828841 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:29.840108 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:29.845845 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:29.863709 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:29.895001 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:29.896025 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:29.896152 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:29.997246 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:30.009621 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:20:30.010986 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:30.084064 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:30.129693 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:30.139287 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:30.166661 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:30.169638 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:30.170378 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:30.196688 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:30.312021 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:20:30.509918 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:30.579469 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:30.588729 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:30.599989 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:30.608412 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:30.620256 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:30.635571 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:30.658486 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:30.675742 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:30.709755 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:30.744233 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:30.828243 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:30.839555 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:20:30.875123 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:30.877655 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:30.884631 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:30.886219 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:30.890805 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:30.916393 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:30.931860 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:31.010035 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:31.033174 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:31.101920 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:31.136358 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:31.144585 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:31.148057 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:31.148505 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:31.167727 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:31.167783 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:31.178078 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:31.201481 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:31.213536 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:31.241741 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:31.250088 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:31.268175 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:31.274920 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:31.440531 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:31.455695 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:31.515309 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:31.537144 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:31.601588 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:31.602216 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:31.632139 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:31.641165 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:31.644280 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:31.682538 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:31.687214 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:31.816644 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:31.866559 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:31.872740 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:31.882484 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:31.898211 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:31.899155 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:31.912433 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:31.962623 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:31.976634 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:31.993386 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:32.035478 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:32.068919 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:32.147404 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:32.148181 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:32.150686 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:32.151301 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:32.160624 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:32.167809 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:32.210973 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:20:32.304273 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:32.318575 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:32.430270 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:32.602610 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:32.604872 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:32.613188 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:32.640730 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:32.646735 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:32.662573 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:32.663030 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:32.723937 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:32.817521 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:32.832798 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:20:32.852309 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:32.862462 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:32.880633 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:32.896369 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:32.903993 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:32.912324 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:33.010304 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:33.081862 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:33.157945 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:33.163856 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:33.164283 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:33.164894 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:33.179652 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:33.187114 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:33.188196 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:33.188452 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:33.203035 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:33.242738 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:33.249294 14684 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.130:35021, user_credentials={real_user=slave}} blocked reactor thread for 46901.4us
W20240503 08:20:33.414973 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:20:33.431298 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:33.616577 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:20:33.625015 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:33.639528 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:33.639513 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:33.648818 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:33.657784 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:33.664757 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:33.701928 14557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240503 08:20:33.721879 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:33.730479 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:33.778191 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:33.814884 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:33.839736 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:33.840106 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:33.870303 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:33.881819 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:33.904655 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:33.930508 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:33.943128 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:33.944100 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:33.965178 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:33.978657 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:33.981631 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:33.990279 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:34.035946 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:34.096774 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:20:34.150447 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:34.181468 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:34.228976 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:34.230738 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:34.246434 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:34.250576 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:34.258136 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:34.265465 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:34.283366 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:34.418932 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:34.469000 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:34.478016 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:34.512410 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:34.633751 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:34.663758 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:34.680035 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:34.685169 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:34.699939 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:34.728649 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:34.733956 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:34.808336 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:34.808595 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:20:34.919763 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:34.942778 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:34.950387 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:20:34.979709 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:35.002816 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:35.003882 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:35.016501 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:35.024516 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:35.155850 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:35.165983 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:35.195027 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 08:20:35.276722 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:35.279461 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:35.280465 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:35.297595 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:35.316228 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:35.322520 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:35.513058 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:20:35.566713 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:35.588888 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:35.616467 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:35.630389 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:35.653048 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:20:35.739985 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:35.745486 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:35.746206 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:35.754412 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:35.777362 14555 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.49.133:38023, user_credentials={real_user=slave}} blocked reactor thread for 61091.4us
I20240503 08:20:35.800591 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:35.813889 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:35.822387 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:35.984817 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:35.987565 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:36.019191 14425 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:20:36.026384 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:36.044598 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:36.088889 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:36.097496 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:36.106717 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:36.119028 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:36.170341 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:36.191504 14686 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:20:36.202329 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:36.247280 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:36.280040 14426 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:36.282399 14426 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:20:36.301317 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:36.311749 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:36.324548 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:36.331459 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:36.333397 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:36.379631 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:36.398348 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:36.477429 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:36.552100 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:36.579901 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:36.598335 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:20:36.618981 14687 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:20:36.679229 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:36.784951 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:36.793814 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:36.803890 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:36.806354 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:36.816354 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:36.834340 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:36.851646 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:36.933131 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:20:36.992242 14426 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:20:37.088701 14686 consensus_peers.cc:462] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:37.090405 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:37.101171 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:37.101161 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:37.112059 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:37.123167 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:37.126886 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:37.169962 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:37.241511 14425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240503 08:20:37.334246 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:20:37.345736 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:37.354784 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:37.368505 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:37.381011 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:37.396504 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:37.398398 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:20:37.428231 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:20:37.487886 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:37.569201 32494 raft_consensus.cc:998] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.137s)
I20240503 08:20:37.569936 32494 consensus_queue.cc:237] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:20:37.572117 14619 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:20:37.572348 16800 raft_consensus.cc:1261] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:20:37.572671   404 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:20:37.572865   402 consensus_queue.cc:1035] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:20:37.575340 32494 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:37.575969 14619 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:37.575953 16800 raft_consensus.cc:2932] T c130eb7ea8a74f4b8d9b79e23af70067 P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:37.580720 14346 catalog_manager.cc:5484] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 10 to 11, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
W20240503 08:20:37.591790 14335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:37.593184 14335 catalog_manager.cc:4573] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c130eb7ea8a74f4b8d9b79e23af70067: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
W20240503 08:20:37.634239 14425 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:20:37.696388 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:20:37.828200 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:37.839062 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:37.842310 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:37.852419 14425 consensus_peers.cc:462] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:37.854456 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:37.860991 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:37.880651 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:37.934829 14817 consensus_peers.cc:462] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:37.986078 14425 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:38.108067   488 raft_consensus.cc:998] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.122s)
I20240503 08:20:38.109056   488 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:38.114472 32741 raft_consensus.cc:998] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.300s)
I20240503 08:20:38.115274 32741 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:38.117630 32494 raft_consensus.cc:998] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.114s)
I20240503 08:20:38.118607 32494 consensus_queue.cc:237] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:38.125977 16403 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:20:38.127096   467 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:20:38.131670 14748 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:20:38.132445   489 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:20:38.135653 16404 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:20:38.137025   499 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:20:38.142339   467 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.148226 16405 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.155289 16404 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.155335 14346 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 reported cstate change: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:20:38.162904 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:20:38.164597   493 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.167313   489 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.168622 14880 raft_consensus.cc:1261] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:20:38.169265   494 consensus_queue.cc:1035] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:20:38.172657 14748 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.176985 14348 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.179188 14342 catalog_manager.cc:5484] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.185060 14880 raft_consensus.cc:2932] T 7aa1031b9e30478597da5ad3708bdff5 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.210762 14490 consensus_queue.cc:237] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
W20240503 08:20:38.213884 14424 consensus_peers.cc:485] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30b13cdf4b234f17872f5d1588915968. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:38.220856 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:38.236671 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:20:38.240469 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:38.241371 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:38.251084 14749 raft_consensus.cc:1261] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:20:38.252076   489 consensus_queue.cc:1035] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:20:38.255813   489 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.259244 14749 raft_consensus.cc:2932] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.262584   490 raft_consensus.cc:998] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.376s)
I20240503 08:20:38.263291   490 consensus_queue.cc:237] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
W20240503 08:20:38.264350 14557 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:38.267653 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:38.267345 14348 catalog_manager.cc:5484] T 30b13cdf4b234f17872f5d1588915968 P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.268165 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b13cdf4b234f17872f5d1588915968 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 08:20:38.288213 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:38.289705 14619 raft_consensus.cc:1261] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:20:38.290323   490 consensus_queue.cc:1035] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:20:38.325701 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:38.330094 14620 consensus_queue.cc:237] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
W20240503 08:20:38.333284 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 562e1ec5c34945298cf457758328057f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:38.333647 16403 raft_consensus.cc:1261] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:20:38.334193   502 consensus_queue.cc:1035] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:20:38.336633   502 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.341488 16403 raft_consensus.cc:2932] T 562e1ec5c34945298cf457758328057f P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.343602 14340 catalog_manager.cc:5484] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 8 to 9, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 1 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.347788 14333 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 562e1ec5c34945298cf457758328057f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:38.350030 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:38.353890   358 raft_consensus.cc:998] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.044s)
I20240503 08:20:38.354826   358 consensus_queue.cc:237] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:38.367566   490 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:38.372557 14620 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:20:38.372864 14749 raft_consensus.cc:1261] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 08:20:38.373112   521 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:20:38.373469   358 consensus_queue.cc:1035] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:20:38.375964   358 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.376478 14620 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.378631 14621 raft_consensus.cc:2932] T da1f165b4a5647a48ad7ff951fb3fbf6 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:38.383695 14348 catalog_manager.cc:5484] T dc9575f64eb741948e51ccf06f48228a P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 12 to 13, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 3 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.384806   488 raft_consensus.cc:998] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.229s)
I20240503 08:20:38.385432   488 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:38.390051 14342 catalog_manager.cc:5484] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 12 to 13, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "95860e22c9474e2d894f04deba8442cc" member_type: VOTER last_known_addr { host: "127.2.49.134" port: 36937 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:38.406977 16403 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:20:38.407583   488 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:20:38.407675 14749 raft_consensus.cc:2932] T dc9575f64eb741948e51ccf06f48228a P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:38.411926   490 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.417758 16403 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.418535 14340 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 7 to 8, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:38.426129 14490 consensus_queue.cc:237] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:38.428819 16403 raft_consensus.cc:1261] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:20:38.429474   489 consensus_queue.cc:1035] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:20:38.434017   489 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
W20240503 08:20:38.434845 14424 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): Couldn't send request to peer b35c4c5d528c4a8f9e754ff9481c3938. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:38.435031 16403 raft_consensus.cc:2932] T a87c5af03f4d47d38bbfbd225c7195d1 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:20:38.438584 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a87c5af03f4d47d38bbfbd225c7195d1 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:38.441262 14340 catalog_manager.cc:5484] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 8 to 9, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
W20240503 08:20:38.496775 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:20:38.588616 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:38.598104 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:38.608438 14335 catalog_manager.cc:4573] TS 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a87c5af03f4d47d38bbfbd225c7195d1: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111)
I20240503 08:20:38.613179 16385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:38.623157 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:38.646303 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:38.646905 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:20:38.726508 14556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240503 08:20:38.734392   502 raft_consensus.cc:998] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.183s)
I20240503 08:20:38.735211   502 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:20:38.737758 16801 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:20:38.738520   579 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:20:38.741357   467 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.741981 16801 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.747826 14347 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.752671 14620 consensus_queue.cc:237] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
W20240503 08:20:38.755259 14555 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): Couldn't send request to peer b35c4c5d528c4a8f9e754ff9481c3938. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:38.756043 16801 raft_consensus.cc:1261] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:20:38.756927   467 consensus_queue.cc:1035] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:20:38.760152   579 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:20:38.765064 14333 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3fea213e5a12452f8b894b9cdb94f9ae with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:38.765616 16801 raft_consensus.cc:2932] T 3fea213e5a12452f8b894b9cdb94f9ae P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:20:38.768707 14347 catalog_manager.cc:5484] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 9 to 10, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New cstate: current_term: 2 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
W20240503 08:20:38.800894 14687 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:38.826531   594 ts_tablet_manager.cc:902] T 30b13cdf4b234f17872f5d1588915968 P 0f8702fad8ea4d22840dab11bb27f7f2: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
F20240503 08:20:38.826939   594 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-12df78f752c34fe3885b235c4a19e325.queue_time_us, tablet-download-pool-12df78f752c34fe3885b235c4a19e325.queue_time_us) (tablet-download-pool-12df78f752c34fe3885b235c4a19e325.run_cpu_time_us, tablet-download-pool-12df78f752c34fe3885b235c4a19e325.run_cpu_time_us) (tablet-download-pool-12df78f752c34fe3885b235c4a19e325.run_wall_time_us, tablet-download-pool-12df78f752c34fe3885b235c4a19e325.run_wall_time_us) (tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.queue_time_us, tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.queue_time_us) (tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_cpu_time_us, tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_cpu_time_us) (tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_wall_time_us, tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_wall_time_us) (tablet-download-pool-30b13cdf4b234f17872f5d1588915968.queue_time_us, tablet-download-pool-30b13cdf4b234f17872f5d1588915968.queue_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.queue_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.queue_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_cpu_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_cpu_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_wall_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.run_wall_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.queue_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.queue_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.run_cpu_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.run_cpu_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.run_wall_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.run_wall_time_us) (tablet-download-pool-5d16c2981213464cbefa44b6ca509182.queue_time_us, tablet-download-pool-5d16c2981213464cbefa44b6ca509182.queue_time_us) (tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_cpu_time_us, tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_cpu_time_us) (tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_wall_time_us, tablet-download-pool-5d16c2981213464cbefa44b6ca509182.run_wall_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.queue_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.queue_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_cpu_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_cpu_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_wall_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_wall_time_us) (tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.queue_time_us, tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.queue_time_us) (tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_cpu_time_us, tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_cpu_time_us) (tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_wall_time_us, tablet-download-pool-7aa1031b9e30478597da5ad3708bdff5.run_wall_time_us) (tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.queue_time_us, tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.queue_time_us) (tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_cpu_time_us, tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_cpu_time_us) (tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_wall_time_us, tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_wall_time_us) (tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.queue_time_us, tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.queue_time_us) (tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_cpu_time_us, tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_cpu_time_us) (tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_wall_time_us, tablet-download-pool-a87c5af03f4d47d38bbfbd225c7195d1.run_wall_time_us) (tablet-download-pool-b5ceefc466f642d08ea96d549a534776.queue_time_us, tablet-download-pool-b5ceefc466f642d08ea96d549a534776.queue_time_us) (tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_cpu_time_us, tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_cpu_time_us) (tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_wall_time_us, tablet-download-pool-b5ceefc466f642d08ea96d549a534776.run_wall_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.queue_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.queue_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_cpu_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_cpu_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_wall_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_wall_time_us) (tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.queue_time_us, tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.queue_time_us) (tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_cpu_time_us, tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_cpu_time_us) (tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_wall_time_us, tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_wall_time_us) (tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.queue_time_us, tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.queue_time_us) (tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_cpu_time_us, tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_cpu_time_us) (tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_wall_time_us, tablet-download-pool-f65ab6dba1124573962e2da58aa1859c.run_wall_time_us) (tablet-download-pool-f7a5221747794a13aa4ed763006ba291.queue_time_us, tablet-download-pool-f7a5221747794a13aa4ed763006ba291.queue_time_us) (tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_cpu_time_us, tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_cpu_time_us) (tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_wall_time_us, tablet-download-pool-f7a5221747794a13aa4ed763006ba291.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f562cdb8dcd  google::LogMessage::Fail() at ??:0
    @     0x7f562cdbcb93  google::LogMessage::SendToLog() at ??:0
    @     0x7f562cdb87cc  google::LogMessage::Flush() at ??:0
    @     0x7f562cdb9f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:20:38.854022   602 ts_tablet_manager.cc:902] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
    @     0x7f562faf9742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f562fac5f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f562fabfcd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f563ef138de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f563ef2bdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f563f056480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:20:38.894594   602 tablet_copy_client.cc:308] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
    @     0x7f563f08ea8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 08:20:38.896991   490 raft_consensus.cc:998] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.168s)
I20240503 08:20:38.897639   490 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
    @     0x7f563f071981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 08:20:38.910722 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 562e1ec5c34945298cf457758328057f from peer 036223b0890d4634acfadf9596f39f80 ({username='slave'} at 127.2.49.136:52119)
I20240503 08:20:38.910945 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 562e1ec5c34945298cf457758328057f from peer 036223b0890d4634acfadf9596f39f80 at {username='slave'} at 127.2.49.136:52119: session id = 036223b0890d4634acfadf9596f39f80-562e1ec5c34945298cf457758328057f
I20240503 08:20:38.914718 32494 raft_consensus.cc:998] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.226s)
I20240503 08:20:38.915019 14641 tablet_copy_source_session.cc:205] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:20:38.915401 32494 consensus_queue.cc:237] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:38.917497   602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 562e1ec5c34945298cf457758328057f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:38.931172   602 tablet_copy_client.cc:783] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 5 data blocks...
I20240503 08:20:38.932073 16801 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:20:38.933039   488 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:20:38.933782   488 raft_consensus.cc:998] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.163s)
I20240503 08:20:38.934562   488 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:20:38.940778   602 tablet_copy_client.cc:647] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:20:38.943104   602 tablet_copy_client.cc:523] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:20:38.948659   602 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: Bootstrap starting.
    @     0x7f563f07170e  std::_Function_handler<>::_M_invoke() at ??:0
I20240503 08:20:38.963513 16800 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:20:38.965308 14880 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:20:38.966336 16403 raft_consensus.cc:1261] T b5ceefc466f642d08ea96d549a534776 P 0f8702fad8ea4d22840dab11bb27f7f2 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:20:38.967382 32494 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:20:38.967808   621 consensus_queue.cc:1035] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f562fac9b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f562facd19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f562facd019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f562faa801e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f5632e2d6db  start_thread at ??:0
    @     0x7f562bc6b71f  clone at ??:0
I20240503 08:20:39.004426 16800 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:38.967687   488 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20240503 08:20:38.990038   490 raft_consensus.cc:998] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.141s)
I20240503 08:20:39.007356   490 consensus_queue.cc:237] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:20:39.015856 14342 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.020607   488 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.039259   602 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:20:39.039671   602 tablet_bootstrap.cc:492] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: Bootstrap complete.
I20240503 08:20:39.039942   602 ts_tablet_manager.cc:1367] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: Time spent bootstrapping tablet: real 0.091s	user 0.027s	sys 0.001s
I20240503 08:20:39.002424   489 raft_consensus.cc:998] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.092s)
I20240503 08:20:39.043435   489 consensus_queue.cc:237] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
I20240503 08:20:39.001662   631 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.044709   602 raft_consensus.cc:348] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:20:39.054684   602 raft_consensus.cc:729] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036223b0890d4634acfadf9596f39f80, State: Initialized, Role: FOLLOWER
I20240503 08:20:39.055052   602 consensus_queue.cc:260] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:20:39.060194   602 ts_tablet_manager.cc:1398] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80: Time spent starting tablet: real 0.020s	user 0.001s	sys 0.003s
I20240503 08:20:39.063907   621 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.064785 14620 raft_consensus.cc:1261] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:20:39.064522 14880 raft_consensus.cc:2932] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.065412   632 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:20:39.066484 16800 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.075621 14342 catalog_manager.cc:5484] T b5ceefc466f642d08ea96d549a534776 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 6 to 7, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.076658   632 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:20:39.079982   638 ts_tablet_manager.cc:902] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:20:39.080920 14620 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:20:39.080992 14347 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 8 to 9, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.082304 14621 raft_consensus.cc:1261] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:20:39.087172 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session 036223b0890d4634acfadf9596f39f80-562e1ec5c34945298cf457758328057f received from {username='slave'} at 127.2.49.136:52119
I20240503 08:20:39.087288 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session 036223b0890d4634acfadf9596f39f80-562e1ec5c34945298cf457758328057f on tablet 562e1ec5c34945298cf457758328057f with peer 036223b0890d4634acfadf9596f39f80
I20240503 08:20:39.089016 14490 consensus_queue.cc:237] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:39.094897   635 consensus_queue.cc:1035] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:20:39.102979 14346 catalog_manager.cc:5484] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 5 to 6, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:20:39.103359 14488 consensus_queue.cc:237] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
W20240503 08:20:39.105618 14424 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075): Couldn't send request to peer b35c4c5d528c4a8f9e754ff9481c3938. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c595bb19d6946b0b1f853064892e946. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:39.106185 16800 raft_consensus.cc:1261] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:20:39.106197 16801 raft_consensus.cc:1261] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:20:39.107173   636 consensus_queue.cc:1035] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:20:39.107712   636 consensus_queue.cc:1035] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:20:39.115285 14489 consensus_queue.cc:237] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:39.116449   636 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:20:39.120560 14620 raft_consensus.cc:1261] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Refusing update from remote peer 129a2ac87d7f416db6d47d79bb1c2b5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:20:39.121150   489 consensus_queue.cc:1035] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:20:39.121377 16800 raft_consensus.cc:2932] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:20:39.126072   635 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.131364   632 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.131932 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6b677894b51f4dc58f157d8f4d632e9a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:39.137042 14620 raft_consensus.cc:2932] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.137706 16800 raft_consensus.cc:2932] T 6b677894b51f4dc58f157d8f4d632e9a P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.143276 14346 catalog_manager.cc:5484] T 5c595bb19d6946b0b1f853064892e946 P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 9 to 10, VOTER b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132) added. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } }
I20240503 08:20:39.146346 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7cc0fe7d81e149d5b82fabfdc11d66b4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:39.146663 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5c595bb19d6946b0b1f853064892e946 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:39.147557 14342 catalog_manager.cc:5484] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 6 to 7, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.149080 14347 catalog_manager.cc:5484] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.175931   631 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:39.179342 14621 raft_consensus.cc:2932] T dd11460051ab40febc2fe6aff58b5028 P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:39.182780 14347 catalog_manager.cc:5484] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f reported cstate change: config changed from index 11 to 12, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 2 leader_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:39.188258   638 tablet_copy_client.cc:308] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:20:39.189450 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet a87c5af03f4d47d38bbfbd225c7195d1 from peer b35c4c5d528c4a8f9e754ff9481c3938 ({username='slave'} at 127.2.49.132:36825)
I20240503 08:20:39.189612 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet a87c5af03f4d47d38bbfbd225c7195d1 from peer b35c4c5d528c4a8f9e754ff9481c3938 at {username='slave'} at 127.2.49.132:36825: session id = b35c4c5d528c4a8f9e754ff9481c3938-a87c5af03f4d47d38bbfbd225c7195d1
I20240503 08:20:39.193710 14507 tablet_copy_source_session.cc:205] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:20:39.195753   638 ts_tablet_manager.cc:1875] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:20:39.203735   638 ts_tablet_manager.cc:1888] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:20:39.204057   638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87c5af03f4d47d38bbfbd225c7195d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:39.207993   638 tablet_copy_client.cc:783] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 5 data blocks...
I20240503 08:20:39.215036   638 tablet_copy_client.cc:647] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 1 WAL segments...
I20240503 08:20:39.219089   638 tablet_copy_client.cc:523] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:20:39.226033 16847 maintenance_manager.cc:419] P 036223b0890d4634acfadf9596f39f80: Scheduling UndoDeltaBlockGCOp(562e1ec5c34945298cf457758328057f): 4583 bytes on disk
I20240503 08:20:39.226917 16738 maintenance_manager.cc:643] P 036223b0890d4634acfadf9596f39f80: UndoDeltaBlockGCOp(562e1ec5c34945298cf457758328057f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 08:20:39.230744 14425 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:39.231066   638 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:20:39.236337   652 ts_tablet_manager.cc:902] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
W20240503 08:20:39.251508 14557 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:39.285452   652 tablet_copy_client.cc:308] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:35021
I20240503 08:20:39.286124 14641 tablet_copy_service.cc:140] P 0d77c4b1f12049dea71b97e86be2b288: Received BeginTabletCopySession request for tablet 3fea213e5a12452f8b894b9cdb94f9ae from peer b35c4c5d528c4a8f9e754ff9481c3938 ({username='slave'} at 127.2.49.132:38085)
I20240503 08:20:39.286289 14641 tablet_copy_service.cc:161] P 0d77c4b1f12049dea71b97e86be2b288: Beginning new tablet copy session on tablet 3fea213e5a12452f8b894b9cdb94f9ae from peer b35c4c5d528c4a8f9e754ff9481c3938 at {username='slave'} at 127.2.49.132:38085: session id = b35c4c5d528c4a8f9e754ff9481c3938-3fea213e5a12452f8b894b9cdb94f9ae
I20240503 08:20:39.289796 14641 tablet_copy_source_session.cc:205] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:20:39.291280   652 ts_tablet_manager.cc:1875] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:20:39.299793   638 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:20:39.300243   638 tablet_bootstrap.cc:492] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap complete.
I20240503 08:20:39.300554   638 ts_tablet_manager.cc:1367] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.070s	user 0.031s	sys 0.011s
I20240503 08:20:39.301329   638 raft_consensus.cc:348] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:39.301544   638 raft_consensus.cc:729] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:20:39.301791   638 consensus_queue.cc:260] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:39.302371   638 ts_tablet_manager.cc:1398] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:20:39.304549 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-a87c5af03f4d47d38bbfbd225c7195d1 received from {username='slave'} at 127.2.49.132:36825
I20240503 08:20:39.304687 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-a87c5af03f4d47d38bbfbd225c7195d1 on tablet a87c5af03f4d47d38bbfbd225c7195d1 with peer b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:20:39.315394   652 ts_tablet_manager.cc:1888] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:20:39.316326   652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea213e5a12452f8b894b9cdb94f9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:39.318802 16800 raft_consensus.cc:1203] T 562e1ec5c34945298cf457758328057f P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 08:20:39.327091   652 tablet_copy_client.cc:783] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 5 data blocks...
I20240503 08:20:39.336670   652 tablet_copy_client.cc:647] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 1 WAL segments...
I20240503 08:20:39.338632   652 tablet_copy_client.cc:523] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:20:39.342626 14426 consensus_peers.cc:485] T dfa711e8c8564bc78846b16d68470cea P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:20:39.350616 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:39.374922   652 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:20:39.379307 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:39.388108   621 raft_consensus.cc:998] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.356s)
I20240503 08:20:39.388774   621 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:20:39.391481 16800 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:20:39.392094 32494 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:20:39.406683 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:20:39.425050 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:39.428550 16801 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.425964 32494 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.436550 14346 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 reported cstate change: config changed from index 7 to 8, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.441635 14878 raft_consensus.cc:1203] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 08:20:39.446717 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:39.464702   358 raft_consensus.cc:998] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Attempting to remove follower 95860e22c9474e2d894f04deba8442cc from the Raft config. Reason: Leader has been unable to successfully communicate with peer 95860e22c9474e2d894f04deba8442cc for more than 300 seconds (300.515s)
I20240503 08:20:39.465577   358 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:39.484355 14749 consensus_queue.cc:237] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:20:39.486654 16800 raft_consensus.cc:1261] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Refusing update from remote peer c0fd1ebec9da48d4b0c1ae0f0ee96452: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:20:39.487308 32494 consensus_queue.cc:1035] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:20:39.489378   621 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.492715 14336 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f19b7ef418974b06bdcc7125e8cf732d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:39.493052 16800 raft_consensus.cc:2932] T f19b7ef418974b06bdcc7125e8cf732d P 036223b0890d4634acfadf9596f39f80 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
I20240503 08:20:39.497716 14346 catalog_manager.cc:5484] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 reported cstate change: config changed from index 8 to 9, VOTER 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } }
W20240503 08:20:39.513834 14817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:20:39.531304   467 raft_consensus.cc:998] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288: Attempting to remove follower 6835ad958e094ce4927135b942a2e8a8 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 6835ad958e094ce4927135b942a2e8a8 for more than 300 seconds (300.019s)
W20240503 08:20:39.533167 14815 connection.cc:535] client connection to 127.2.49.133:38023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:39.533123   467 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } }
W20240503 08:20:39.534618 14555 connection.cc:535] client connection to 127.2.49.133:38023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:39.536151 14685 connection.cc:535] client connection to 127.2.49.133:38023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:39.536494 14424 connection.cc:535] client connection to 127.2.49.133:38023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:39.537434 14749 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:20:39.537707 14424 consensus_peers.cc:485] T dd11460051ab40febc2fe6aff58b5028 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.537811 14424 consensus_peers.cc:462] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111)
W20240503 08:20:39.537875 14424 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.537940 14424 consensus_peers.cc:485] T a87c5af03f4d47d38bbfbd225c7195d1 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.538002 14424 consensus_peers.cc:485] T 7cc0fe7d81e149d5b82fabfdc11d66b4 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.538410 14685 consensus_peers.cc:485] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.538529 14685 consensus_peers.cc:485] T b5ceefc466f642d08ea96d549a534776 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.538612 14685 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.538689 14685 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:39.539341   687 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:20:39.540472 14815 consensus_peers.cc:485] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.540583 14815 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:39.540549 14489 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 4 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
W20240503 08:20:39.541357 32742 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:20:39.535375 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.133:38023 (local address 127.2.49.130:51839)
0503 08:20:39.535684 (+   309us) negotiation.cc:107] Waiting for socket to connect
0503 08:20:39.535707 (+    23us) client_negotiation.cc:174] Beginning negotiation
0503 08:20:39.535863 (+   156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:20:39.541105 (+  5242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":177}
W20240503 08:20:39.541724 14555 consensus_peers.cc:485] T f7a5221747794a13aa4ed763006ba291 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:39.541869 14555 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:39.541179   467 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240503 08:20:39.542236   358 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:39.547664 14749 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:39.549522 14347 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 9 to 10, VOTER 95860e22c9474e2d894f04deba8442cc (127.2.49.134) evicted. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
I20240503 08:20:39.567373 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(a87c5af03f4d47d38bbfbd225c7195d1): 4584 bytes on disk
I20240503 08:20:39.568006 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(a87c5af03f4d47d38bbfbd225c7195d1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:20:39.570209 14878 consensus_queue.cc:237] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:20:39.573776 14749 raft_consensus.cc:1261] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Refusing update from remote peer b35c4c5d528c4a8f9e754ff9481c3938: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:20:39.574115 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55067e08fb694e90b6366d41cc962095. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:39.574936   358 consensus_queue.cc:1035] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:20:39.576870   693 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 4 LEADER]: Committing config change with OpId 4.17: config changed from index 15 to 17, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:39.578817   652 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:20:39.578919   687 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:20:39.579315   652 tablet_bootstrap.cc:492] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap complete.
I20240503 08:20:39.579722   652 ts_tablet_manager.cc:1367] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.205s	user 0.016s	sys 0.016s
I20240503 08:20:39.580550   652 raft_consensus.cc:348] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:39.580855   652 raft_consensus.cc:729] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:20:39.581141   652 consensus_queue.cc:260] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:39.581924   652 ts_tablet_manager.cc:1398] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:20:39.582671 14641 tablet_copy_service.cc:342] P 0d77c4b1f12049dea71b97e86be2b288: Request end of tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-3fea213e5a12452f8b894b9cdb94f9ae received from {username='slave'} at 127.2.49.132:38085
I20240503 08:20:39.582821 14641 tablet_copy_service.cc:433] P 0d77c4b1f12049dea71b97e86be2b288: ending tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-3fea213e5a12452f8b894b9cdb94f9ae on tablet 3fea213e5a12452f8b894b9cdb94f9ae with peer b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:20:39.584337   638 ts_tablet_manager.cc:902] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: Initiating tablet copy from peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829)
I20240503 08:20:39.584829 14334 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 55067e08fb694e90b6366d41cc962095 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:20:39.586336 14749 raft_consensus.cc:2932] T 55067e08fb694e90b6366d41cc962095 P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:20:39.587196 14346 catalog_manager.cc:5484] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 reported cstate change: config changed from index 10 to 11, VOTER 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129) added. New cstate: current_term: 1 leader_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } }
I20240503 08:20:39.587307 14489 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 4 FOLLOWER]: Committing config change with OpId 4.17: config changed from index 15 to 17, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:39.596490 14347 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 15 to 17, VOTER 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135) evicted. New cstate: current_term: 4 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } }
I20240503 08:20:39.597061   638 tablet_copy_client.cc:308] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:37829
I20240503 08:20:39.597743 14507 tablet_copy_service.cc:140] P 129a2ac87d7f416db6d47d79bb1c2b5f: Received BeginTabletCopySession request for tablet 5c595bb19d6946b0b1f853064892e946 from peer b35c4c5d528c4a8f9e754ff9481c3938 ({username='slave'} at 127.2.49.132:36825)
I20240503 08:20:39.597892 14507 tablet_copy_service.cc:161] P 129a2ac87d7f416db6d47d79bb1c2b5f: Beginning new tablet copy session on tablet 5c595bb19d6946b0b1f853064892e946 from peer b35c4c5d528c4a8f9e754ff9481c3938 at {username='slave'} at 127.2.49.132:36825: session id = b35c4c5d528c4a8f9e754ff9481c3938-5c595bb19d6946b0b1f853064892e946
I20240503 08:20:39.601400 14507 tablet_copy_source_session.cc:205] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:20:39.603538   638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c595bb19d6946b0b1f853064892e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:39.604830 14621 consensus_queue.cc:237] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:20:39.607218 14489 raft_consensus.cc:1261] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 4 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 4 index: 18. (index mismatch)
W20240503 08:20:39.607273 14556 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e86f4f3a01048e69f553c6557d79b71. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:39.607888   693 consensus_queue.cc:1035] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240503 08:20:39.611136   693 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 [term 4 LEADER]: Committing config change with OpId 4.18: config changed from index 17 to 18, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.613751 14488 raft_consensus.cc:2932] T 4e86f4f3a01048e69f553c6557d79b71 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 4 FOLLOWER]: Committing config change with OpId 4.18: config changed from index 17 to 18, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.616079 14333 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4e86f4f3a01048e69f553c6557d79b71 with cas_config_opid_index 17: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 08:20:39.620092 14336 catalog_manager.cc:4573] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f19b7ef418974b06bdcc7125e8cf732d: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
I20240503 08:20:39.622054 14347 catalog_manager.cc:5484] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: config changed from index 17 to 18, VOTER 036223b0890d4634acfadf9596f39f80 (127.2.49.136) added. New cstate: current_term: 4 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } }
I20240503 08:20:39.623757   638 tablet_copy_client.cc:783] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 5 data blocks...
I20240503 08:20:39.631513   638 tablet_copy_client.cc:647] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Starting download of 1 WAL segments...
I20240503 08:20:39.633234   638 tablet_copy_client.cc:523] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:20:39.636531   638 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap starting.
I20240503 08:20:39.651656   638 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:20:39.652032   638 tablet_bootstrap.cc:492] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: Bootstrap complete.
I20240503 08:20:39.652287   638 ts_tablet_manager.cc:1367] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent bootstrapping tablet: real 0.016s	user 0.016s	sys 0.000s
I20240503 08:20:39.652915   638 raft_consensus.cc:348] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:39.653127   638 raft_consensus.cc:729] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c4c5d528c4a8f9e754ff9481c3938, State: Initialized, Role: FOLLOWER
I20240503 08:20:39.653373   638 consensus_queue.cc:260] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:39.653838   638 ts_tablet_manager.cc:1398] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:20:39.654647 14507 tablet_copy_service.cc:342] P 129a2ac87d7f416db6d47d79bb1c2b5f: Request end of tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-5c595bb19d6946b0b1f853064892e946 received from {username='slave'} at 127.2.49.132:36825
I20240503 08:20:39.654778 14507 tablet_copy_service.cc:433] P 129a2ac87d7f416db6d47d79bb1c2b5f: ending tablet copy session b35c4c5d528c4a8f9e754ff9481c3938-5c595bb19d6946b0b1f853064892e946 on tablet 5c595bb19d6946b0b1f853064892e946 with peer b35c4c5d528c4a8f9e754ff9481c3938
I20240503 08:20:39.753602 14878 raft_consensus.cc:1203] T 3fea213e5a12452f8b894b9cdb94f9ae P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240503 08:20:39.820655 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(3fea213e5a12452f8b894b9cdb94f9ae): 4923 bytes on disk
I20240503 08:20:39.821322 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(3fea213e5a12452f8b894b9cdb94f9ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 08:20:39.822312 14927 maintenance_manager.cc:419] P b35c4c5d528c4a8f9e754ff9481c3938: Scheduling UndoDeltaBlockGCOp(5c595bb19d6946b0b1f853064892e946): 4838 bytes on disk
I20240503 08:20:39.822755 14818 maintenance_manager.cc:643] P b35c4c5d528c4a8f9e754ff9481c3938: UndoDeltaBlockGCOp(5c595bb19d6946b0b1f853064892e946) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240503 08:20:39.887018 14556 consensus_peers.cc:485] T 76a216295a8a4feaa2d3326035879073 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:20:39.978427 16781 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:39.984539 14468 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 08:20:39.999070 14424 consensus_peers.cc:462] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111)
I20240503 08:20:40.018601 14599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:40.036480   744 ts_tablet_manager.cc:902] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: Initiating tablet copy from peer b35c4c5d528c4a8f9e754ff9481c3938 (127.2.49.132:35075)
I20240503 08:20:40.046026 14860 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:20:40.081177   744 tablet_copy_client.cc:308] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.132:35075
I20240503 08:20:40.082028 14901 tablet_copy_service.cc:140] P b35c4c5d528c4a8f9e754ff9481c3938: Received BeginTabletCopySession request for tablet 55067e08fb694e90b6366d41cc962095 from peer 129a2ac87d7f416db6d47d79bb1c2b5f ({username='slave'} at 127.2.49.129:42359)
I20240503 08:20:40.082239 14901 tablet_copy_service.cc:161] P b35c4c5d528c4a8f9e754ff9481c3938: Beginning new tablet copy session on tablet 55067e08fb694e90b6366d41cc962095 from peer 129a2ac87d7f416db6d47d79bb1c2b5f at {username='slave'} at 127.2.49.129:42359: session id = 129a2ac87d7f416db6d47d79bb1c2b5f-55067e08fb694e90b6366d41cc962095
I20240503 08:20:40.086227 14901 tablet_copy_source_session.cc:205] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:20:40.088078   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55067e08fb694e90b6366d41cc962095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:40.090065 14730 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:40.096308   744 tablet_copy_client.cc:783] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 5 data blocks...
I20240503 08:20:40.105533   744 tablet_copy_client.cc:647] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Starting download of 1 WAL segments...
I20240503 08:20:40.107614   744 tablet_copy_client.cc:523] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:20:40.112358   744 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap starting.
I20240503 08:20:40.132437   744 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:20:40.132799   744 tablet_bootstrap.cc:492] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: Bootstrap complete.
I20240503 08:20:40.133097   744 ts_tablet_manager.cc:1367] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.010s
I20240503 08:20:40.133805   744 raft_consensus.cc:348] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:20:40.133991   744 raft_consensus.cc:729] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a2ac87d7f416db6d47d79bb1c2b5f, State: Initialized, Role: FOLLOWER
I20240503 08:20:40.134267   744 consensus_queue.cc:260] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }
I20240503 08:20:40.134781   744 ts_tablet_manager.cc:1398] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:20:40.135613 14901 tablet_copy_service.cc:342] P b35c4c5d528c4a8f9e754ff9481c3938: Request end of tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-55067e08fb694e90b6366d41cc962095 received from {username='slave'} at 127.2.49.129:42359
I20240503 08:20:40.135735 14901 tablet_copy_service.cc:433] P b35c4c5d528c4a8f9e754ff9481c3938: ending tablet copy session 129a2ac87d7f416db6d47d79bb1c2b5f-55067e08fb694e90b6366d41cc962095 on tablet 55067e08fb694e90b6366d41cc962095 with peer 129a2ac87d7f416db6d47d79bb1c2b5f
I20240503 08:20:40.151165   751 ts_tablet_manager.cc:902] T 4e86f4f3a01048e69f553c6557d79b71 P 036223b0890d4634acfadf9596f39f80: Initiating tablet copy from peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021)
F20240503 08:20:40.151515   751 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0780fb979eee4bae98a014d9721c9441.queue_time_us, tablet-download-pool-0780fb979eee4bae98a014d9721c9441.queue_time_us) (tablet-download-pool-0780fb979eee4bae98a014d9721c9441.run_cpu_time_us, tablet-download-pool-0780fb979eee4bae98a014d9721c9441.run_cpu_time_us) (tablet-download-pool-0780fb979eee4bae98a014d9721c9441.run_wall_time_us, tablet-download-pool-0780fb979eee4bae98a014d9721c9441.run_wall_time_us) (tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.queue_time_us, tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.queue_time_us) (tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_cpu_time_us, tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_cpu_time_us) (tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_wall_time_us, tablet-download-pool-169bd9ea6772420abf757dd55cf44d31.run_wall_time_us) (tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.queue_time_us, tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.queue_time_us) (tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_cpu_time_us, tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_cpu_time_us) (tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_wall_time_us, tablet-download-pool-1d6c920958214148b3c3d5ac995e99e2.run_wall_time_us) (tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.queue_time_us, tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.queue_time_us) (tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_cpu_time_us, tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_cpu_time_us) (tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_wall_time_us, tablet-download-pool-3fea213e5a12452f8b894b9cdb94f9ae.run_wall_time_us) (tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.queue_time_us, tablet-download-pool-4e86f4f3a01048e69f553c6557d79b71.queue_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.queue_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.queue_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.run_cpu_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.run_cpu_time_us) (tablet-download-pool-562e1ec5c34945298cf457758328057f.run_wall_time_us, tablet-download-pool-562e1ec5c34945298cf457758328057f.run_wall_time_us) (tablet-download-pool-5c595bb19d6946b0b1f853064892e946.queue_time_us, tablet-download-pool-5c595bb19d6946b0b1f853064892e946.queue_time_us) (tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_cpu_time_us, tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_cpu_time_us) (tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_wall_time_us, tablet-download-pool-5c595bb19d6946b0b1f853064892e946.run_wall_time_us) (tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.queue_time_us, tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.queue_time_us) (tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_cpu_time_us, tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_cpu_time_us) (tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_wall_time_us, tablet-download-pool-6b677894b51f4dc58f157d8f4d632e9a.run_wall_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.queue_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.queue_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_cpu_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_cpu_time_us) (tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_wall_time_us, tablet-download-pool-72a949e76ab5477680a4f9912e9672e9.run_wall_time_us) (tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.queue_time_us, tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.queue_time_us) (tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_cpu_time_us, tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_cpu_time_us) (tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_wall_time_us, tablet-download-pool-88404b9cdf574cb0bca1bb49348fb4cd.run_wall_time_us) (tablet-download-pool-897a701b880641b08816f2d1a254ef03.queue_time_us, tablet-download-pool-897a701b880641b08816f2d1a254ef03.queue_time_us) (tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_cpu_time_us, tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_cpu_time_us) (tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_wall_time_us, tablet-download-pool-897a701b880641b08816f2d1a254ef03.run_wall_time_us) (tablet-download-pool-b19df3ae9f2b44a3818855f259962fff.queue_time_us, tablet-download-pool-b19df3ae9f2b44a3818855f259962fff.queue_time_us) (tablet-download-pool-b19df3ae9f2b44a3818855f259962fff.run_cpu_time_us, tablet-download-pool-b19df3ae9f2b44a3818855f259962fff.run_cpu_time_us) (tablet-download-pool-b19df3ae9f2b44a3818855f259962fff.run_wall_time_us, tablet-download-pool-b19df3ae9f2b44a3818855f259962fff.run_wall_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.queue_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.queue_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_cpu_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_cpu_time_us) (tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_wall_time_us, tablet-download-pool-c130eb7ea8a74f4b8d9b79e23af70067.run_wall_time_us) (tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.queue_time_us, tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.queue_time_us) (tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_cpu_time_us, tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_cpu_time_us) (tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_wall_time_us, tablet-download-pool-dd11460051ab40febc2fe6aff58b5028.run_wall_time_us) (tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.queue_time_us, tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.queue_time_us) (tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_cpu_time_us, tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_cpu_time_us) (tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_wall_time_us, tablet-download-pool-f19b7ef418974b06bdcc7125e8cf732d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 08:20:40.153591 14878 raft_consensus.cc:1203] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
    @     0x7fc6cc999dcd  google::LogMessage::Fail() at ??:0
    @     0x7fc6cc99db93  google::LogMessage::SendToLog() at ??:0
    @     0x7fc6cc9997cc  google::LogMessage::Flush() at ??:0
    @     0x7fc6cc99af59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:20:40.167428 14536 maintenance_manager.cc:419] P 129a2ac87d7f416db6d47d79bb1c2b5f: Scheduling UndoDeltaBlockGCOp(55067e08fb694e90b6366d41cc962095): 4583 bytes on disk
    @     0x7fc6cf6da742  kudu::TraceMetrics::InternName() at ??:0
I20240503 08:20:40.168294 14427 maintenance_manager.cc:643] P 129a2ac87d7f416db6d47d79bb1c2b5f: UndoDeltaBlockGCOp(55067e08fb694e90b6366d41cc962095) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
    @     0x7fc6cf6a6f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7fc6cf6a0cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7fc6deaf48de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7fc6deb0cdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7fc6dec37480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7fc6dec6fa8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7fc6dec52981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7fc6dec5270e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fc6cf6aab75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7fc6cf6ae19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7fc6cf6ae019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fc6cf68901e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fc6d2a0e6db  start_thread at ??:0
    @     0x7fc6cb84c71f  clone at ??:0
W20240503 08:20:40.258558 14425 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 6835ad958e094ce4927135b942a2e8a8 (127.2.49.135:46299): Couldn't send request to peer 6835ad958e094ce4927135b942a2e8a8. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:20:40.273759 32494 raft_consensus.cc:482] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0f8702fad8ea4d22840dab11bb27f7f2)
I20240503 08:20:40.273972 32494 raft_consensus.cc:504] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:40.274735 32494 leader_election.cc:288] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021), 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
W20240503 08:20:40.275758 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240503 08:20:40.276223 14621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "0d77c4b1f12049dea71b97e86be2b288" is_pre_election: true
W20240503 08:20:40.276866 14685 leader_election.cc:334] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111)
I20240503 08:20:40.276979 14685 leader_election.cc:302] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 0d77c4b1f12049dea71b97e86be2b288, 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:20:40.277261 32494 raft_consensus.cc:2726] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:20:40.354432   773 raft_consensus.cc:482] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0f8702fad8ea4d22840dab11bb27f7f2)
I20240503 08:20:40.354676   773 raft_consensus.cc:504] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:40.355677   773 leader_election.cc:288] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
W20240503 08:20:40.357352 14555 leader_election.cc:334] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111)
I20240503 08:20:40.363492 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" is_pre_election: true
I20240503 08:20:40.363757 14749 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 2.
I20240503 08:20:40.364256 14557 leader_election.cc:302] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:20:40.364614   773 raft_consensus.cc:2781] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:20:40.364724   773 raft_consensus.cc:482] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0f8702fad8ea4d22840dab11bb27f7f2)
I20240503 08:20:40.364817   773 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:20:40.368312   773 raft_consensus.cc:504] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:40.369182   773 leader_election.cc:288] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 3 election: Requested vote from peers 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023), c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663)
I20240503 08:20:40.370033 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65ab6dba1124573962e2da58aa1859c" candidate_uuid: "0d77c4b1f12049dea71b97e86be2b288" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452"
I20240503 08:20:40.370301 14749 raft_consensus.cc:3037] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:20:40.370738 14555 leader_election.cc:334] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111)
I20240503 08:20:40.373848 14749 raft_consensus.cc:2454] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d77c4b1f12049dea71b97e86be2b288 in term 3.
I20240503 08:20:40.374693 14557 leader_election.cc:302] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d77c4b1f12049dea71b97e86be2b288, c0fd1ebec9da48d4b0c1ae0f0ee96452; no voters: 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:20:40.375269   773 raft_consensus.cc:2781] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:20:40.375452   773 raft_consensus.cc:686] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [term 3 LEADER]: Becoming Leader. State: Replica: 0d77c4b1f12049dea71b97e86be2b288, State: Running, Role: LEADER
I20240503 08:20:40.375754   773 consensus_queue.cc:237] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } }
I20240503 08:20:40.379966 14348 catalog_manager.cc:5484] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 reported cstate change: term changed from 2 to 3, leader changed from 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133) to 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130). New cstate: current_term: 3 leader_uuid: "0d77c4b1f12049dea71b97e86be2b288" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } }
W20240503 08:20:40.464715 14556 connection.cc:535] client connection to 127.2.49.136:37641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:40.464723 14425 connection.cc:535] client connection to 127.2.49.136:37641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:40.465370 14816 connection.cc:535] client connection to 127.2.49.136:37641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:40.466406 14556 consensus_peers.cc:485] T 562e1ec5c34945298cf457758328057f P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.466569 14556 consensus_peers.cc:485] T 3fea213e5a12452f8b894b9cdb94f9ae P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.466670 14556 consensus_peers.cc:462] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111)
W20240503 08:20:40.466864 14425 consensus_peers.cc:485] T 6b677894b51f4dc58f157d8f4d632e9a P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.466922 14816 consensus_peers.cc:485] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:40.510233 14488 raft_consensus.cc:1203] T 55067e08fb694e90b6366d41cc962095 P 129a2ac87d7f416db6d47d79bb1c2b5f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240503 08:20:40.528082 14686 consensus_peers.cc:485] T 1d6c920958214148b3c3d5ac995e99e2 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.572280 14556 consensus_peers.cc:462] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111)
W20240503 08:20:40.580472 14424 consensus_peers.cc:462] T 30b13cdf4b234f17872f5d1588915968 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111)
W20240503 08:20:40.587821 14686 consensus_peers.cc:485] T c130eb7ea8a74f4b8d9b79e23af70067 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.605947 14686 consensus_peers.cc:485] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.643642 14686 consensus_peers.cc:485] T f19b7ef418974b06bdcc7125e8cf732d P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.644153 14816 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.648401 14336 catalog_manager.cc:4573] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dc9575f64eb741948e51ccf06f48228a: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:40.652050 14816 consensus_peers.cc:485] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.670804 14426 consensus_peers.cc:485] T da1f165b4a5647a48ad7ff951fb3fbf6 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:40.711027 14425 consensus_peers.cc:485] T 5c595bb19d6946b0b1f853064892e946 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:40.743391 14556 consensus_peers.cc:485] T b19df3ae9f2b44a3818855f259962fff P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Couldn't send request to peer 036223b0890d4634acfadf9596f39f80. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:40.774046 16852 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 08:14:28.939440 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:28.939805 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:28.955252 16855 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:31.831620 16855 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:14:33.970538 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:33.970853 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:33.971009 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:33.971185 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:33.971279 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:33.971413 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:33.971494 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:33.971581 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:33.971649 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:33.971743 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:33.972260 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:33.972378 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:33.972476 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:33.972574 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:33.972661 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 0f8702fad8ea4d22840dab11bb27f7f2
I20240503 08:14:34.839043 16855 rebalancer_tool.cc:1074] tablet f65ab6dba1124573962e2da58aa1859c: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:35.473539 16855 rebalancer_tool.cc:1074] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:36.022151 16855 rebalancer_tool.cc:1074] tablet 3fea213e5a12452f8b894b9cdb94f9ae: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:36.105492 16855 rebalancer_tool.cc:1084] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '036223b0890d4634acfadf9596f39f80' move ignored: Illegal state: tablet 88404b9cdf574cb0bca1bb49348fb4cd: replica 036223b0890d4634acfadf9596f39f80 is already present
I20240503 08:14:36.106170 16855 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:38.385155 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.404613 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.404748 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:38.404928 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:38.408313 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.408490 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.410511 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:38.410645 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 64eb775ebfdc4fad97f1290c92213ff7: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:38.410733 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:38.410826 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.411013 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:38.413348 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:38.413522 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table afbaaee2198d4c62a28b88392a8b6e06: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:38.413666 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 95860e22c9474e2d894f04deba8442cc
I20240503 08:14:38.988337 16855 rebalancer_tool.cc:1074] tablet 169bd9ea6772420abf757dd55cf44d31: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:39.112253 16855 rebalancer_tool.cc:1074] tablet 30b13cdf4b234f17872f5d1588915968: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:14:39.676764 16855 rebalancer_tool.cc:1074] tablet f7a5221747794a13aa4ed763006ba291: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:14:40.322077 16855 rebalancer_tool.cc:1074] tablet dfa711e8c8564bc78846b16d68470cea: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:14:40.707665 16855 rebalancer_tool.cc:1074] tablet 562e1ec5c34945298cf457758328057f: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:14:41.045328 16855 rebalancer_tool.cc:1084] tablet 12df78f752c34fe3885b235c4a19e325: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '036223b0890d4634acfadf9596f39f80' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 } } peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } }
I20240503 08:14:41.045506 16855 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:14:42.549903 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:42.550293 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:42.550581 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6815d5df4efd45d59a2db67226231af0: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 036223b0890d4634acfadf9596f39f80
W20240503 08:14:42.550734 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b120ce671eaf40d099d0c2d7e71d6cbe: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:42.550875 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5224e66c2cf144369fb29255cdd25306: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:42.551028 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f9617d8cfb394c7d8239ea6325ea98db: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 6835ad958e094ce4927135b942a2e8a8
W20240503 08:14:42.551327 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d5bb4e27f85346a0953339b3915769bc: could not find any suitable replica to move from server b35c4c5d528c4a8f9e754ff9481c3938 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:42.551451 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3d6b5b0c5d2848d989f291fdcbf71d6f: could not find any suitable replica to move from server 0d77c4b1f12049dea71b97e86be2b288 to server 95860e22c9474e2d894f04deba8442cc
W20240503 08:14:42.551573 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0f8702fad8ea4d22840dab11bb27f7f2
W20240503 08:14:42.551692 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7184b9ddb57b427cb18c0e1bbf24e3a1: could not find any suitable replica to move from server c0fd1ebec9da48d4b0c1ae0f0ee96452 to server 036223b0890d4634acfadf9596f39f80
I20240503 08:14:42.877869 16855 rebalancer_tool.cc:1074] tablet 897a701b880641b08816f2d1a254ef03: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:14:43.360033 16855 rebalancer_tool.cc:1074] tablet 0780fb979eee4bae98a014d9721c9441: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:43.475803 16855 rebalancer_tool.cc:1074] tablet b19df3ae9f2b44a3818855f259962fff: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:14:44.022743 16855 rebalancer_tool.cc:1074] tablet f19b7ef418974b06bdcc7125e8cf732d: '0d77c4b1f12049dea71b97e86be2b288' -> '036223b0890d4634acfadf9596f39f80' move scheduled
I20240503 08:14:44.183912 16855 rebalancer_tool.cc:1074] tablet c130eb7ea8a74f4b8d9b79e23af70067: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move scheduled
I20240503 08:14:44.286837 16855 rebalancer_tool.cc:1074] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:14:49.056965 16855 rebalancer_tool.cc:1123] tablet b19df3ae9f2b44a3818855f259962fff: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move is abandoned: Not found: Server with UUID 129a2ac87d7f416db6d47d79bb1c2b5f not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d77c4b1f12049dea71b97e86be2b288" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 35021 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } }
I20240503 08:14:58.333981 16855 rebalancer_tool.cc:1136] tablet 55067e08fb694e90b6366d41cc962095: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:14:58.404196 16855 rebalancer_tool.cc:1136] tablet 88404b9cdf574cb0bca1bb49348fb4cd: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:06.664608 16855 rebalancer_tool.cc:1136] tablet 3fea213e5a12452f8b894b9cdb94f9ae: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:06.782887 16855 rebalancer_tool.cc:1136] tablet 30b13cdf4b234f17872f5d1588915968: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:06.854367 16855 rebalancer_tool.cc:1136] tablet f7a5221747794a13aa4ed763006ba291: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '6835ad958e094ce4927135b942a2e8a8' move completed: OK
I20240503 08:15:06.913951 16855 rebalancer_tool.cc:1074] tablet 5c595bb19d6946b0b1f853064892e946: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:06.990394 16855 rebalancer_tool.cc:1074] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move scheduled
I20240503 08:15:10.546588 16855 rebalancer_tool.cc:1136] tablet c130eb7ea8a74f4b8d9b79e23af70067: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:10.901558 16855 rebalancer_tool.cc:1136] tablet 76a216295a8a4feaa2d3326035879073: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '' move completed: OK
I20240503 08:15:10.996171 16855 rebalancer_tool.cc:1136] tablet 897a701b880641b08816f2d1a254ef03: 'b35c4c5d528c4a8f9e754ff9481c3938' -> '' move completed: OK
I20240503 08:15:11.021636 16855 rebalancer_tool.cc:1136] tablet 562e1ec5c34945298cf457758328057f: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:11.155180 16855 rebalancer_tool.cc:1136] tablet f19b7ef418974b06bdcc7125e8cf732d: '0d77c4b1f12049dea71b97e86be2b288' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:11.407997 16855 rebalancer_tool.cc:1136] tablet dfa711e8c8564bc78846b16d68470cea: '0d77c4b1f12049dea71b97e86be2b288' -> '95860e22c9474e2d894f04deba8442cc' move completed: OK
I20240503 08:15:11.528405 16855 rebalancer_tool.cc:1136] tablet f65ab6dba1124573962e2da58aa1859c: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: OK
I20240503 08:15:11.596503 16855 rebalancer_tool.cc:1136] tablet 0780fb979eee4bae98a014d9721c9441: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:13.922260 16855 rebalancer_tool.cc:1074] tablet 6b677894b51f4dc58f157d8f4d632e9a: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:14.174430 16855 rebalancer_tool.cc:1074] tablet 4e86f4f3a01048e69f553c6557d79b71: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:14.427975 16855 rebalancer_tool.cc:1074] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:16.627766 16855 rebalancer_tool.cc:1136] tablet 5c595bb19d6946b0b1f853064892e946: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:16.808262 16855 rebalancer_tool.cc:1136] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:17.011883 16855 rebalancer_tool.cc:1136] tablet 6b677894b51f4dc58f157d8f4d632e9a: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
I20240503 08:15:17.247817 16855 rebalancer_tool.cc:1136] tablet 169bd9ea6772420abf757dd55cf44d31: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '036223b0890d4634acfadf9596f39f80' move completed: OK
I20240503 08:15:17.540586 16855 rebalancer_tool.cc:1074] tablet dd11460051ab40febc2fe6aff58b5028: '0d77c4b1f12049dea71b97e86be2b288' -> '' move scheduled
I20240503 08:15:17.629634 16855 rebalancer_tool.cc:1074] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:17.804351 16855 rebalancer_tool.cc:1074] tablet 1c687520c0a349fd8438dacbd5011001: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:17.970364 16855 rebalancer_tool.cc:1136] tablet 1c687520c0a349fd8438dacbd5011001: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:18.013432 16855 rebalancer_tool.cc:1136] tablet 7cc0fe7d81e149d5b82fabfdc11d66b4: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:21.497205 16855 rebalancer_tool.cc:1136] tablet 4e86f4f3a01048e69f553c6557d79b71: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:21.551240 16855 rebalancer_tool.cc:1074] tablet dc9575f64eb741948e51ccf06f48228a: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move scheduled
I20240503 08:15:28.610682 16855 rebalancer_tool.cc:1136] tablet dc9575f64eb741948e51ccf06f48228a: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '' move completed: OK
I20240503 08:15:28.667147 16855 rebalancer_tool.cc:1136] tablet dd11460051ab40febc2fe6aff58b5028: '0d77c4b1f12049dea71b97e86be2b288' -> '' move completed: OK
W20240503 08:15:30.114071 16855 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e5e40744d3943e7b71fb29a47a92384: could not find any suitable replica to move from server 129a2ac87d7f416db6d47d79bb1c2b5f to server 0d77c4b1f12049dea71b97e86be2b288
I20240503 08:15:30.478410 16855 rebalancer_tool.cc:1074] tablet dd11460051ab40febc2fe6aff58b5028: '036223b0890d4634acfadf9596f39f80' -> '0d77c4b1f12049dea71b97e86be2b288' move scheduled
I20240503 08:15:31.301347 16855 rebalancer_tool.cc:1074] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '6835ad958e094ce4927135b942a2e8a8' -> '0d77c4b1f12049dea71b97e86be2b288' move scheduled
I20240503 08:15:35.807674 16855 rebalancer_tool.cc:1136] tablet da1f165b4a5647a48ad7ff951fb3fbf6: '6835ad958e094ce4927135b942a2e8a8' -> '0d77c4b1f12049dea71b97e86be2b288' move completed: OK
I20240503 08:15:35.964803 16855 rebalancer_tool.cc:1136] tablet dd11460051ab40febc2fe6aff58b5028: '036223b0890d4634acfadf9596f39f80' -> '0d77c4b1f12049dea71b97e86be2b288' move completed: OK
I20240503 08:15:37.466360 16855 rebalancer_tool.cc:1074] tablet c130eb7ea8a74f4b8d9b79e23af70067: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:15:37.971634 16855 rebalancer_tool.cc:1074] tablet 72a949e76ab5477680a4f9912e9672e9: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move scheduled
I20240503 08:15:38.405067 16855 rebalancer_tool.cc:1074] tablet dc9575f64eb741948e51ccf06f48228a: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '95860e22c9474e2d894f04deba8442cc' move scheduled
I20240503 08:15:39.096431 16855 rebalancer_tool.cc:1074] tablet dd11460051ab40febc2fe6aff58b5028: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move scheduled
I20240503 08:15:41.088706 16855 rebalancer_tool.cc:1136] tablet 72a949e76ab5477680a4f9912e9672e9: '129a2ac87d7f416db6d47d79bb1c2b5f' -> '0f8702fad8ea4d22840dab11bb27f7f2' move completed: Incomplete: tablet 72a949e76ab5477680a4f9912e9672e9, TS 129a2ac87d7f416db6d47d79bb1c2b5f -> TS 0f8702fad8ea4d22840dab11bb27f7f2 move failed, target replica disappeared
I20240503 08:20:38.455677 16855 rebalancer_tool.cc:1136] tablet c130eb7ea8a74f4b8d9b79e23af70067: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move completed: Incomplete: tablet c130eb7ea8a74f4b8d9b79e23af70067, TS 0d77c4b1f12049dea71b97e86be2b288 -> TS 6835ad958e094ce4927135b942a2e8a8 move failed, target replica disappeared
I20240503 08:20:38.739337 16855 rebalancer_tool.cc:1136] tablet dc9575f64eb741948e51ccf06f48228a: 'c0fd1ebec9da48d4b0c1ae0f0ee96452' -> '95860e22c9474e2d894f04deba8442cc' move completed: Incomplete: tablet dc9575f64eb741948e51ccf06f48228a, TS c0fd1ebec9da48d4b0c1ae0f0ee96452 -> TS 95860e22c9474e2d894f04deba8442cc move failed, target replica disappeared
W20240503 08:20:39.538122   667 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:20:39.356426 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.49.133:38023 (local address 127.0.0.1:58632)
0503 08:20:39.356589 (+   163us) negotiation.cc:107] Waiting for socket to connect
0503 08:20:39.356610 (+    21us) client_negotiation.cc:174] Beginning negotiation
0503 08:20:39.356712 (+   102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:20:39.536674 (+179962us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
I20240503 08:20:39.835099 16855 rebalancer_tool.cc:1136] tablet dd11460051ab40febc2fe6aff58b5028: '0d77c4b1f12049dea71b97e86be2b288' -> '6835ad958e094ce4927135b942a2e8a8' move completed: Incomplete: tablet dd11460051ab40febc2fe6aff58b5028, TS 0d77c4b1f12049dea71b97e86be2b288 -> TS 6835ad958e094ce4927135b942a2e8a8 move failed, target replica disappeared
Illegal state: tablet server 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): unacceptable health status UNAVAILABLE

E20240503 08:20:40.788964  2246 external_mini_cluster.cc:943] Process with UUID 0f8702fad8ea4d22840dab11bb27f7f2 has crashed
E20240503 08:20:40.789155  2246 external_mini_cluster.cc:943] Process with UUID 95860e22c9474e2d894f04deba8442cc has crashed
E20240503 08:20:40.789260  2246 external_mini_cluster.cc:943] Process with UUID 6835ad958e094ce4927135b942a2e8a8 has crashed
E20240503 08:20:40.789340  2246 external_mini_cluster.cc:943] Process with UUID 036223b0890d4634acfadf9596f39f80 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 4
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240503 08:20:40.790913  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14408
W20240503 08:20:40.792987 14426 consensus_peers.cc:485] T 1c687520c0a349fd8438dacbd5011001 P 129a2ac87d7f416db6d47d79bb1c2b5f -> Peer 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): Couldn't send request to peer 95860e22c9474e2d894f04deba8442cc. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:20:40.799022 14528 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:40.883280 14749 raft_consensus.cc:1261] T f65ab6dba1124573962e2da58aa1859c P c0fd1ebec9da48d4b0c1ae0f0ee96452 [term 3 FOLLOWER]: Refusing update from remote peer 0d77c4b1f12049dea71b97e86be2b288: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:20:40.884207   773 consensus_queue.cc:1035] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fd1ebec9da48d4b0c1ae0f0ee96452" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:20:40.891991 14555 consensus_peers.cc:485] T f65ab6dba1124573962e2da58aa1859c P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.112571 14556 consensus_peers.cc:462] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.49.136:37641: connect: Connection refused (error 111)
I20240503 08:20:41.144095 14338 maintenance_manager.cc:419] P 309eed6422ad4e71bcf73f518dc42423: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9242 bytes on disk
I20240503 08:20:41.149700 14337 maintenance_manager.cc:643] P 309eed6422ad4e71bcf73f518dc42423: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s	user 0.005s	sys 0.000s Metrics: {"dirs.queue_time_us":410,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":119,"thread_start_us":445,"threads_started":1}
I20240503 08:20:41.216056  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14408
W20240503 08:20:41.274642 14687 connection.cc:535] client connection to 127.2.49.129:37829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:41.275350  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14539
W20240503 08:20:41.275478 14557 connection.cc:535] client connection to 127.2.49.129:37829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.275832 14557 connection.cc:535] server connection from 127.2.49.129:60669 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.276118 14687 consensus_peers.cc:485] T 897a701b880641b08816f2d1a254ef03 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276392 14817 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276533 14817 consensus_peers.cc:485] T 72a949e76ab5477680a4f9912e9672e9 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276623 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.276939 14557 consensus_peers.cc:485] T 4e86f4f3a01048e69f553c6557d79b71 P 0d77c4b1f12049dea71b97e86be2b288 -> Peer 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829): Couldn't send request to peer 129a2ac87d7f416db6d47d79bb1c2b5f. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:37829: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:41.282294 14661 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:41.642069  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14539
W20240503 08:20:41.645644 14815 consensus_peers.cc:485] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:41.672734 14685 consensus_peers.cc:485] T 7aa1031b9e30478597da5ad3708bdff5 P c0fd1ebec9da48d4b0c1ae0f0ee96452 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:41.700901 14336 connection.cc:535] server connection from 127.2.49.130:44725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.700883 14814 connection.cc:535] client connection to 127.2.49.130:35021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:20:41.700910 14816 connection.cc:535] server connection from 127.2.49.130:33947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:41.701182  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14669
W20240503 08:20:41.702270 14814 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:35021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:41.707424 14790 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:20:41.735582 14814 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:35021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:41.787412 14336 catalog_manager.cc:4573] TS 95860e22c9474e2d894f04deba8442cc (127.2.49.134:36937): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 55067e08fb694e90b6366d41cc962095: Network error: Client connection negotiation failed: client connection to 127.2.49.134:36937: connect: Connection refused (error 111)
W20240503 08:20:41.815070 14815 consensus_peers.cc:485] T 5d16c2981213464cbefa44b6ca509182 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023): Couldn't send request to peer 0f8702fad8ea4d22840dab11bb27f7f2. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.133:38023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:20:41.945169 14814 consensus_peers.cc:485] T 88404b9cdf574cb0bca1bb49348fb4cd P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer 0d77c4b1f12049dea71b97e86be2b288 (127.2.49.130:35021): Couldn't send request to peer 0d77c4b1f12049dea71b97e86be2b288. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.130:35021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:42.074105  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14669
W20240503 08:20:42.133157 14817 connection.cc:535] client connection to 127.2.49.131:45663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:42.133484  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14799
W20240503 08:20:42.134572 14817 consensus_peers.cc:485] T 0780fb979eee4bae98a014d9721c9441 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.155920 14817 consensus_peers.cc:485] T dc9575f64eb741948e51ccf06f48228a P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.156056 14817 consensus_peers.cc:485] T 12df78f752c34fe3885b235c4a19e325 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.156121 14817 consensus_peers.cc:485] T 169bd9ea6772420abf757dd55cf44d31 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:20:42.156193 14817 consensus_peers.cc:485] T 55067e08fb694e90b6366d41cc962095 P b35c4c5d528c4a8f9e754ff9481c3938 -> Peer c0fd1ebec9da48d4b0c1ae0f0ee96452 (127.2.49.131:45663): Couldn't send request to peer c0fd1ebec9da48d4b0c1ae0f0ee96452. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:45663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:20:42.156744 14917 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:42.511545   797 raft_consensus.cc:482] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 129a2ac87d7f416db6d47d79bb1c2b5f)
I20240503 08:20:42.511729   797 raft_consensus.cc:504] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "0f8702fad8ea4d22840dab11bb27f7f2" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38023 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:42.512372   797 leader_election.cc:288] T a87c5af03f4d47d38bbfbd225c7195d1 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 0f8702fad8ea4d22840dab11bb27f7f2 (127.2.49.133:38023)
I20240503 08:20:42.513053   687 raft_consensus.cc:482] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 129a2ac87d7f416db6d47d79bb1c2b5f)
I20240503 08:20:42.513206   687 raft_consensus.cc:504] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "129a2ac87d7f416db6d47d79bb1c2b5f" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 37829 } } peers { permanent_uuid: "036223b0890d4634acfadf9596f39f80" member_type: VOTER last_known_addr { host: "127.2.49.136" port: 37641 } } peers { permanent_uuid: "b35c4c5d528c4a8f9e754ff9481c3938" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 35075 } }
I20240503 08:20:42.513726  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14799
I20240503 08:20:42.513799   687 leader_election.cc:288] T 5c595bb19d6946b0b1f853064892e946 P b35c4c5d528c4a8f9e754ff9481c3938 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 129a2ac87d7f416db6d47d79bb1c2b5f (127.2.49.129:37829), 036223b0890d4634acfadf9596f39f80 (127.2.49.136:37641)
I20240503 08:20:42.564114  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 14318
I20240503 08:20:42.570080 14376 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:20:42.917398 14335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.135:46299: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240503 08:20:42.917982  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 14318
2024-05-03T08:20:42Z chronyd exiting
I20240503 08:20:42.992394  2246 test_util.cc:175] -----------------------------------------------
I20240503 08:20:42.992499  2246 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723866985054-2246-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (397161 ms)
[----------] 1 test from ConcurrentRebalancersTest (397161 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240503 08:20:42.994817  2246 test_util.cc:268] Using random seed: 1699739740
I20240503 08:20:42.996657  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:20:42.996726  2246 ts_itest-base.cc:113] --------------
I20240503 08:20:42.996793  2246 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:20:42.996867  2246 ts_itest-base.cc:115] 1 replicas per TS
I20240503 08:20:42.996932  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:20:43Z Disabled control of system clock
I20240503 08:20:43.051645  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:37189
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:37189
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 08:20:43.447628   818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:43.447993   818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:43.448133   818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:43.455832   818 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:43.455972   818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:20:43.456023   818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:43.456121   818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:20:43.456188   818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:20:43.465605   818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:37189
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:37189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:43.467002   818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:43.468974   818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:43.480142   824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:43.480778   825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:43.480978   827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:43.481715   818 server_base.cc:1034] running on GCE node
I20240503 08:20:43.482566   818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:43.484059   818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:43.485267   818 hybrid_clock.cc:648] HybridClock initialized: now 1714724443485218 us; error 41 us; skew 500 ppm
I20240503 08:20:43.485585   818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:43.493777   818 webserver.cc:460] Webserver started at http://127.2.49.190:43013/ using document root <none> and password file <none>
I20240503 08:20:43.494652   818 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:43.494766   818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:43.495064   818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:43.497843   818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d274a4a7b9e44116abdebe167556b009"
format_stamp: "Formatted at 2024-05-03 08:20:43 on dist-test-slave-dz87"
I20240503 08:20:43.498718   818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d274a4a7b9e44116abdebe167556b009"
format_stamp: "Formatted at 2024-05-03 08:20:43 on dist-test-slave-dz87"
I20240503 08:20:43.504320   818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:20:43.507850   833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:43.509275   818 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I20240503 08:20:43.509462   818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d274a4a7b9e44116abdebe167556b009"
format_stamp: "Formatted at 2024-05-03 08:20:43 on dist-test-slave-dz87"
I20240503 08:20:43.509681   818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:43.566800   818 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:43.567704   818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:43.568029   818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:43.587070   818 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:37189
I20240503 08:20:43.587090   884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:37189 every 8 connection(s)
I20240503 08:20:43.588529   818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:20:43.592247   885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:43.596905  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 818
I20240503 08:20:43.597163  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:20:43.603245   885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009: Bootstrap starting.
I20240503 08:20:43.606410   885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:43.607466   885 log.cc:824] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009: Log is configured to *not* fsync() on all Append() calls
I20240503 08:20:43.610491   885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009: No bootstrap required, opened a new log
I20240503 08:20:43.618522   885 raft_consensus.cc:348] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274a4a7b9e44116abdebe167556b009" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 37189 } }
I20240503 08:20:43.619103   885 raft_consensus.cc:374] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:43.619238   885 raft_consensus.cc:729] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d274a4a7b9e44116abdebe167556b009, State: Initialized, Role: FOLLOWER
I20240503 08:20:43.620117   885 consensus_queue.cc:260] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274a4a7b9e44116abdebe167556b009" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 37189 } }
I20240503 08:20:43.620416   885 raft_consensus.cc:388] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:43.620589   885 raft_consensus.cc:482] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:43.620805   885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:43.624015   885 raft_consensus.cc:504] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274a4a7b9e44116abdebe167556b009" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 37189 } }
I20240503 08:20:43.624680   885 leader_election.cc:302] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d274a4a7b9e44116abdebe167556b009; no voters: 
I20240503 08:20:43.625381   885 leader_election.cc:288] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:43.625545   890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:43.626214   890 raft_consensus.cc:686] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [term 1 LEADER]: Becoming Leader. State: Replica: d274a4a7b9e44116abdebe167556b009, State: Running, Role: LEADER
I20240503 08:20:43.626749   890 consensus_queue.cc:237] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274a4a7b9e44116abdebe167556b009" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 37189 } }
I20240503 08:20:43.627399   885 sys_catalog.cc:564] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:20:43.630014   891 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d274a4a7b9e44116abdebe167556b009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274a4a7b9e44116abdebe167556b009" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 37189 } } }
I20240503 08:20:43.630507   891 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [sys.catalog]: This master's current role is: LEADER
I20240503 08:20:43.631382   898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:20:43.631198   892 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d274a4a7b9e44116abdebe167556b009. Latest consensus state: current_term: 1 leader_uuid: "d274a4a7b9e44116abdebe167556b009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274a4a7b9e44116abdebe167556b009" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 37189 } } }
I20240503 08:20:43.636443   892 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009 [sys.catalog]: This master's current role is: LEADER
I20240503 08:20:43.637149   898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:20:43.647595   898 catalog_manager.cc:1347] Generated new cluster ID: 57f1c71b2ab8476daf7ddc44f6fd5686
I20240503 08:20:43.647742   898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:20:43.660554   898 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:20:43.662387   898 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:20:43.679589   898 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d274a4a7b9e44116abdebe167556b009: Generated new TSK 0
I20240503 08:20:43.680536   898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:20:43.695158  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:20:44.057255   909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:44.057569   909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:44.057698   909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:44.065505   909 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:44.065650   909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:44.065811   909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:20:44.075542   909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:44.076967   909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:44.078842   909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:44.090982   914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:44.091169   917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:44.091710   915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:44.092388   909 server_base.cc:1034] running on GCE node
I20240503 08:20:44.093043   909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:44.094038   909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:44.095194   909 hybrid_clock.cc:648] HybridClock initialized: now 1714724444095135 us; error 50 us; skew 500 ppm
I20240503 08:20:44.095530   909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:44.098018   909 webserver.cc:460] Webserver started at http://127.2.49.129:37507/ using document root <none> and password file <none>
I20240503 08:20:44.098824   909 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:44.098946   909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:44.099261   909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:44.101958   909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ed80600cd2b54ab08512e5301673272b"
format_stamp: "Formatted at 2024-05-03 08:20:44 on dist-test-slave-dz87"
I20240503 08:20:44.102870   909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ed80600cd2b54ab08512e5301673272b"
format_stamp: "Formatted at 2024-05-03 08:20:44 on dist-test-slave-dz87"
I20240503 08:20:44.108392   909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:20:44.112069   923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:44.113576   909 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:20:44.113785   909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ed80600cd2b54ab08512e5301673272b"
format_stamp: "Formatted at 2024-05-03 08:20:44 on dist-test-slave-dz87"
I20240503 08:20:44.113999   909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:44.153326   909 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:44.154310   909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:44.154596   909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:44.155668   909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:44.157267   909 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:44.157369   909 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:44.157485   909 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:44.157559   909 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:44.190758   909 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:43229
I20240503 08:20:44.190833  1035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:43229 every 8 connection(s)
I20240503 08:20:44.192127   909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:20:44.199240  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 909
I20240503 08:20:44.199512  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:20:44.203676  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:20:44.207468  1036 heartbeater.cc:347] Connected to a master server at 127.2.49.190:37189
I20240503 08:20:44.207839  1036 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:44.208818  1036 heartbeater.cc:511] Master 127.2.49.190:37189 requested a full tablet report, sending...
I20240503 08:20:44.211344   850 ts_manager.cc:194] Registered new tserver with Master: ed80600cd2b54ab08512e5301673272b (127.2.49.129:43229)
I20240503 08:20:44.213591   850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:42807
W20240503 08:20:44.556655  1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:44.556989  1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:44.557107  1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:44.564829  1040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:44.564968  1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:44.565145  1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:20:44.574471  1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:44.575852  1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:44.577610  1040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:44.589915  1046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:44.590457  1048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:44.590721  1045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:44.590994  1040 server_base.cc:1034] running on GCE node
I20240503 08:20:44.591717  1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:44.592860  1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:44.594094  1040 hybrid_clock.cc:648] HybridClock initialized: now 1714724444594013 us; error 73 us; skew 500 ppm
I20240503 08:20:44.594514  1040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:44.597254  1040 webserver.cc:460] Webserver started at http://127.2.49.130:42183/ using document root <none> and password file <none>
I20240503 08:20:44.598194  1040 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:44.598323  1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:44.598671  1040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:44.602128  1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4b12f470f64f4e29b3da57f18eee2586"
format_stamp: "Formatted at 2024-05-03 08:20:44 on dist-test-slave-dz87"
I20240503 08:20:44.603163  1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4b12f470f64f4e29b3da57f18eee2586"
format_stamp: "Formatted at 2024-05-03 08:20:44 on dist-test-slave-dz87"
I20240503 08:20:44.609802  1040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20240503 08:20:44.613277  1054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:44.614686  1040 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:20:44.614837  1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4b12f470f64f4e29b3da57f18eee2586"
format_stamp: "Formatted at 2024-05-03 08:20:44 on dist-test-slave-dz87"
I20240503 08:20:44.615021  1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:44.640506  1040 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:44.641430  1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:44.641739  1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:44.642836  1040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:44.644438  1040 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:44.644529  1040 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:44.644672  1040 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:44.644762  1040 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:44.678138  1040 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:37085
I20240503 08:20:44.678238  1166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:37085 every 8 connection(s)
I20240503 08:20:44.679595  1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:20:44.681761  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 1040
I20240503 08:20:44.681952  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:20:44.685716  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:20:44.694262  1167 heartbeater.cc:347] Connected to a master server at 127.2.49.190:37189
I20240503 08:20:44.694656  1167 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:44.695659  1167 heartbeater.cc:511] Master 127.2.49.190:37189 requested a full tablet report, sending...
I20240503 08:20:44.697350   850 ts_manager.cc:194] Registered new tserver with Master: 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130:37085)
I20240503 08:20:44.698289   850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:34467
W20240503 08:20:45.038831  1170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:45.039151  1170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:45.039310  1170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:45.047125  1170 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:45.047291  1170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:45.047423  1170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:20:45.056777  1170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:45.058118  1170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:45.060005  1170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:45.072712  1176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:45.072660  1175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:45.073302  1178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:45.074102  1170 server_base.cc:1034] running on GCE node
I20240503 08:20:45.074728  1170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:45.075701  1170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:45.076906  1170 hybrid_clock.cc:648] HybridClock initialized: now 1714724445076839 us; error 54 us; skew 500 ppm
I20240503 08:20:45.077224  1170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:45.079762  1170 webserver.cc:460] Webserver started at http://127.2.49.131:44555/ using document root <none> and password file <none>
I20240503 08:20:45.080571  1170 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:45.080682  1170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:45.081013  1170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:45.086473  1170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "749f12d2084f43c5af05a839def4fe02"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:45.087363  1170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "749f12d2084f43c5af05a839def4fe02"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:45.092988  1170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:20:45.096563  1184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:45.097940  1170 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:20:45.098098  1170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "749f12d2084f43c5af05a839def4fe02"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:45.098335  1170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:45.117539  1170 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:45.118697  1170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:45.119041  1170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:45.120193  1170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:45.121888  1170 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:45.122002  1170 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:45.122131  1170 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:45.122244  1170 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240503 08:20:45.155741  1170 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:38039
I20240503 08:20:45.155792  1296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:38039 every 8 connection(s)
I20240503 08:20:45.157084  1170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:20:45.162037  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 1170
I20240503 08:20:45.162251  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:20:45.165398  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:20:45.171097  1297 heartbeater.cc:347] Connected to a master server at 127.2.49.190:37189
I20240503 08:20:45.171449  1297 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:45.172232  1297 heartbeater.cc:511] Master 127.2.49.190:37189 requested a full tablet report, sending...
I20240503 08:20:45.173866   850 ts_manager.cc:194] Registered new tserver with Master: 749f12d2084f43c5af05a839def4fe02 (127.2.49.131:38039)
I20240503 08:20:45.175081   850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:37635
I20240503 08:20:45.217535  1036 heartbeater.cc:503] Master 127.2.49.190:37189 was elected leader, sending a full tablet report...
W20240503 08:20:45.517527  1300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:45.517861  1300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:45.517982  1300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:45.525943  1300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:45.526116  1300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:45.526295  1300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:20:45.535990  1300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:37189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:45.537395  1300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:45.539259  1300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:45.551784  1305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:45.552007  1308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:45.552577  1306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:45.553270  1300 server_base.cc:1034] running on GCE node
I20240503 08:20:45.553887  1300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:45.554899  1300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:45.556107  1300 hybrid_clock.cc:648] HybridClock initialized: now 1714724445556053 us; error 45 us; skew 500 ppm
I20240503 08:20:45.556434  1300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:45.558913  1300 webserver.cc:460] Webserver started at http://127.2.49.132:34979/ using document root <none> and password file <none>
I20240503 08:20:45.559667  1300 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:45.559778  1300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:45.560093  1300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:45.562762  1300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "337e9dba83ca4221aa7dceb63124a5fe"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:45.563577  1300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "337e9dba83ca4221aa7dceb63124a5fe"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:45.569089  1300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240503 08:20:45.572713  1314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:45.574129  1300 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:20:45.574350  1300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "337e9dba83ca4221aa7dceb63124a5fe"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:45.574558  1300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:45.593636  1300 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:45.594805  1300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:45.595136  1300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:45.596372  1300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:45.598028  1300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:45.598121  1300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:45.598229  1300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:45.598285  1300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:45.632586  1300 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:33473
I20240503 08:20:45.632647  1426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:33473 every 8 connection(s)
I20240503 08:20:45.633962  1300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:20:45.641764  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 1300
I20240503 08:20:45.641952  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:20:45.647598  1427 heartbeater.cc:347] Connected to a master server at 127.2.49.190:37189
I20240503 08:20:45.647922  1427 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:45.648725  1427 heartbeater.cc:511] Master 127.2.49.190:37189 requested a full tablet report, sending...
I20240503 08:20:45.650117   850 ts_manager.cc:194] Registered new tserver with Master: 337e9dba83ca4221aa7dceb63124a5fe (127.2.49.132:33473)
I20240503 08:20:45.651010   850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:44619
I20240503 08:20:45.658224  2246 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:20:45.675236   850 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36422:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 08:20:45.700956  1167 heartbeater.cc:503] Master 127.2.49.190:37189 was elected leader, sending a full tablet report...
I20240503 08:20:45.704587  1362 tablet_service.cc:1467] Processing CreateTablet for tablet 6296a73ffe264abd952c6462a2d78e1f (DEFAULT_TABLE table=TestTable [id=f776bebd82be4a7cb35396be2fb16a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:45.706034  1362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6296a73ffe264abd952c6462a2d78e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:45.716334  1441 tablet_bootstrap.cc:492] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Bootstrap starting.
I20240503 08:20:45.719472  1441 tablet_bootstrap.cc:654] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:45.720496  1441 log.cc:824] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Log is configured to *not* fsync() on all Append() calls
I20240503 08:20:45.722994  1441 tablet_bootstrap.cc:492] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: No bootstrap required, opened a new log
I20240503 08:20:45.723263  1441 ts_tablet_manager.cc:1367] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 08:20:45.729606  1441 raft_consensus.cc:348] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:45.729995  1441 raft_consensus.cc:374] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:45.730091  1441 raft_consensus.cc:729] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337e9dba83ca4221aa7dceb63124a5fe, State: Initialized, Role: FOLLOWER
I20240503 08:20:45.730731  1441 consensus_queue.cc:260] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:45.730942  1441 raft_consensus.cc:388] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:45.731068  1441 raft_consensus.cc:482] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:45.731226  1441 raft_consensus.cc:3037] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:45.733438  1441 raft_consensus.cc:504] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:45.733923  1441 leader_election.cc:302] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 337e9dba83ca4221aa7dceb63124a5fe; no voters: 
I20240503 08:20:45.734612  1441 leader_election.cc:288] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:45.734784  1443 raft_consensus.cc:2781] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:45.735384  1443 raft_consensus.cc:686] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 LEADER]: Becoming Leader. State: Replica: 337e9dba83ca4221aa7dceb63124a5fe, State: Running, Role: LEADER
I20240503 08:20:45.735752  1427 heartbeater.cc:503] Master 127.2.49.190:37189 was elected leader, sending a full tablet report...
I20240503 08:20:45.735972  1443 consensus_queue.cc:237] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:45.736440  1441 ts_tablet_manager.cc:1398] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.004s
I20240503 08:20:45.741212   850 catalog_manager.cc:5484] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 337e9dba83ca4221aa7dceb63124a5fe (127.2.49.132). New cstate: current_term: 1 leader_uuid: "337e9dba83ca4221aa7dceb63124a5fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } } }
I20240503 08:20:45.750588  2246 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:20:45.752707  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ed80600cd2b54ab08512e5301673272b to finish bootstrapping
I20240503 08:20:45.762369  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4b12f470f64f4e29b3da57f18eee2586 to finish bootstrapping
I20240503 08:20:45.771203  2246 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 749f12d2084f43c5af05a839def4fe02 to finish bootstrapping
I20240503 08:20:45.780299  2246 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 337e9dba83ca4221aa7dceb63124a5fe to finish bootstrapping
I20240503 08:20:45.788573  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 1170
I20240503 08:20:45.789605  1292 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:45.873096  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 1170
I20240503 08:20:45.891216  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 1300
I20240503 08:20:45.892390  1422 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:45.981544  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 1300
I20240503 08:20:49.751469   849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36422:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:20:49.772660  1102 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa2cea1d8d04e16ac720bb7cd65c44f (DEFAULT_TABLE table=rebalance_test_table_0 [id=13199d93c4d24b4297c709340791a5b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:49.772701   970 tablet_service.cc:1467] Processing CreateTablet for tablet 298a0c656d0e4dca90fc5036126b1e71 (DEFAULT_TABLE table=rebalance_test_table_0 [id=13199d93c4d24b4297c709340791a5b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:49.772724   971 tablet_service.cc:1467] Processing CreateTablet for tablet 63c88dbb9b40439795b7c189b4095d63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=13199d93c4d24b4297c709340791a5b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:49.774247  1102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa2cea1d8d04e16ac720bb7cd65c44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:49.774384   970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 298a0c656d0e4dca90fc5036126b1e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:49.774853   971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c88dbb9b40439795b7c189b4095d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:49.786836  1462 tablet_bootstrap.cc:492] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b: Bootstrap starting.
I20240503 08:20:49.787096  1463 tablet_bootstrap.cc:492] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586: Bootstrap starting.
I20240503 08:20:49.790786  1463 tablet_bootstrap.cc:654] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:49.790786  1462 tablet_bootstrap.cc:654] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:49.792248  1462 log.cc:824] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:20:49.792255  1463 log.cc:824] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586: Log is configured to *not* fsync() on all Append() calls
I20240503 08:20:49.795471  1463 tablet_bootstrap.cc:492] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586: No bootstrap required, opened a new log
I20240503 08:20:49.795665  1462 tablet_bootstrap.cc:492] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b: No bootstrap required, opened a new log
I20240503 08:20:49.795822  1463 ts_tablet_manager.cc:1367] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240503 08:20:49.795931  1462 ts_tablet_manager.cc:1367] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.003s
I20240503 08:20:49.805406  1462 raft_consensus.cc:348] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.805408  1463 raft_consensus.cc:348] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:49.805887  1462 raft_consensus.cc:374] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:49.805887  1463 raft_consensus.cc:374] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:49.806005  1463 raft_consensus.cc:729] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Initialized, Role: FOLLOWER
I20240503 08:20:49.806005  1462 raft_consensus.cc:729] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Initialized, Role: FOLLOWER
I20240503 08:20:49.806797  1462 consensus_queue.cc:260] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.806797  1463 consensus_queue.cc:260] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:49.807067  1463 raft_consensus.cc:388] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:49.807067  1462 raft_consensus.cc:388] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:49.807228  1462 raft_consensus.cc:482] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:49.807229  1463 raft_consensus.cc:482] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:49.807404  1463 raft_consensus.cc:3037] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:49.807404  1462 raft_consensus.cc:3037] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:49.810536  1462 raft_consensus.cc:504] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.810536  1463 raft_consensus.cc:504] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:49.811177  1463 leader_election.cc:302] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b12f470f64f4e29b3da57f18eee2586; no voters: 
I20240503 08:20:49.811177  1462 leader_election.cc:302] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed80600cd2b54ab08512e5301673272b; no voters: 
I20240503 08:20:49.811985  1462 leader_election.cc:288] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:49.812084  1467 raft_consensus.cc:2781] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:49.812510  1463 leader_election.cc:288] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:49.812747  1466 raft_consensus.cc:2781] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:49.816110  1466 raft_consensus.cc:686] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [term 1 LEADER]: Becoming Leader. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Running, Role: LEADER
I20240503 08:20:49.821812  1462 ts_tablet_manager.cc:1398] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b: Time spent starting tablet: real 0.026s	user 0.018s	sys 0.006s
I20240503 08:20:49.821610  1466 consensus_queue.cc:237] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:49.822455  1462 tablet_bootstrap.cc:492] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b: Bootstrap starting.
I20240503 08:20:49.824952  1462 tablet_bootstrap.cc:654] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:49.825917  1467 raft_consensus.cc:686] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [term 1 LEADER]: Becoming Leader. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Running, Role: LEADER
I20240503 08:20:49.826646  1467 consensus_queue.cc:237] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.828472  1463 ts_tablet_manager.cc:1398] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586: Time spent starting tablet: real 0.033s	user 0.026s	sys 0.008s
I20240503 08:20:49.831877  1462 tablet_bootstrap.cc:492] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b: No bootstrap required, opened a new log
I20240503 08:20:49.832139  1462 ts_tablet_manager.cc:1367] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.001s
I20240503 08:20:49.832139   849 catalog_manager.cc:5484] T 0fa2cea1d8d04e16ac720bb7cd65c44f P 4b12f470f64f4e29b3da57f18eee2586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "4b12f470f64f4e29b3da57f18eee2586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } } }
I20240503 08:20:49.833284  1462 raft_consensus.cc:348] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.833528  1462 raft_consensus.cc:374] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:49.833624  1462 raft_consensus.cc:729] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Initialized, Role: FOLLOWER
I20240503 08:20:49.833911  1462 consensus_queue.cc:260] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.834123  1462 raft_consensus.cc:388] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:49.834239  1462 raft_consensus.cc:482] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:49.834357  1462 raft_consensus.cc:3037] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:49.836833   850 catalog_manager.cc:5484] T 298a0c656d0e4dca90fc5036126b1e71 P ed80600cd2b54ab08512e5301673272b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed80600cd2b54ab08512e5301673272b (127.2.49.129). New cstate: current_term: 1 leader_uuid: "ed80600cd2b54ab08512e5301673272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } } }
I20240503 08:20:49.837769  1462 raft_consensus.cc:504] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.838043  1462 leader_election.cc:302] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed80600cd2b54ab08512e5301673272b; no voters: 
I20240503 08:20:49.838263  1462 leader_election.cc:288] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:49.838460  1467 raft_consensus.cc:2781] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:49.838707  1467 raft_consensus.cc:686] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [term 1 LEADER]: Becoming Leader. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Running, Role: LEADER
I20240503 08:20:49.838905  1462 ts_tablet_manager.cc:1398] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 08:20:49.838958  1467 consensus_queue.cc:237] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:49.843364   849 catalog_manager.cc:5484] T 63c88dbb9b40439795b7c189b4095d63 P ed80600cd2b54ab08512e5301673272b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed80600cd2b54ab08512e5301673272b (127.2.49.129). New cstate: current_term: 1 leader_uuid: "ed80600cd2b54ab08512e5301673272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } } }
I20240503 08:20:50.375814   849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36422:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:20:50.384764  1102 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1f90ec066648f6a289c2505354c04b (DEFAULT_TABLE table=rebalance_test_table_1 [id=f8597723c2454f9693a331ecc5f46999]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:50.385164   971 tablet_service.cc:1467] Processing CreateTablet for tablet d9ecca63cc22418abc3bd2a6e7f2bce8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f8597723c2454f9693a331ecc5f46999]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:50.385293  1102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1f90ec066648f6a289c2505354c04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:50.385440  1101 tablet_service.cc:1467] Processing CreateTablet for tablet b8dfa0ed2ace4dc690f4292fca4a6369 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f8597723c2454f9693a331ecc5f46999]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:50.385540   971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ecca63cc22418abc3bd2a6e7f2bce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:50.385828  1101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dfa0ed2ace4dc690f4292fca4a6369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:50.392226  1502 tablet_bootstrap.cc:492] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b: Bootstrap starting.
I20240503 08:20:50.393749  1502 tablet_bootstrap.cc:654] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:50.395385  1503 tablet_bootstrap.cc:492] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586: Bootstrap starting.
I20240503 08:20:50.395859  1502 tablet_bootstrap.cc:492] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b: No bootstrap required, opened a new log
I20240503 08:20:50.396014  1502 ts_tablet_manager.cc:1367] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:20:50.396840  1502 raft_consensus.cc:348] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.397039  1502 raft_consensus.cc:374] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:50.397154  1502 raft_consensus.cc:729] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Initialized, Role: FOLLOWER
I20240503 08:20:50.397217  1503 tablet_bootstrap.cc:654] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:50.397454  1502 consensus_queue.cc:260] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.397693  1502 raft_consensus.cc:388] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:50.397789  1502 raft_consensus.cc:482] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:50.397908  1502 raft_consensus.cc:3037] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:50.399297  1503 tablet_bootstrap.cc:492] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586: No bootstrap required, opened a new log
I20240503 08:20:50.399453  1503 ts_tablet_manager.cc:1367] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 08:20:50.400409  1503 raft_consensus.cc:348] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.400604  1503 raft_consensus.cc:374] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:50.400540  1502 raft_consensus.cc:504] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.400699  1503 raft_consensus.cc:729] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Initialized, Role: FOLLOWER
I20240503 08:20:50.400791  1502 leader_election.cc:302] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed80600cd2b54ab08512e5301673272b; no voters: 
I20240503 08:20:50.400998  1502 leader_election.cc:288] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:50.400964  1503 consensus_queue.cc:260] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.401158  1503 raft_consensus.cc:388] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:50.401130  1467 raft_consensus.cc:2781] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:50.401249  1503 raft_consensus.cc:482] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:50.401382  1503 raft_consensus.cc:3037] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:50.401983  1502 ts_tablet_manager.cc:1398] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240503 08:20:50.403523  1503 raft_consensus.cc:504] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.403741  1503 leader_election.cc:302] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b12f470f64f4e29b3da57f18eee2586; no voters: 
I20240503 08:20:50.403956  1467 raft_consensus.cc:686] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [term 1 LEADER]: Becoming Leader. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Running, Role: LEADER
I20240503 08:20:50.404256  1503 leader_election.cc:288] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:50.404399  1508 raft_consensus.cc:2781] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:50.404268  1467 consensus_queue.cc:237] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.408109  1508 raft_consensus.cc:686] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [term 1 LEADER]: Becoming Leader. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Running, Role: LEADER
I20240503 08:20:50.408295   849 catalog_manager.cc:5484] T d9ecca63cc22418abc3bd2a6e7f2bce8 P ed80600cd2b54ab08512e5301673272b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed80600cd2b54ab08512e5301673272b (127.2.49.129). New cstate: current_term: 1 leader_uuid: "ed80600cd2b54ab08512e5301673272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } } }
I20240503 08:20:50.408808  1503 ts_tablet_manager.cc:1398] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.005s
I20240503 08:20:50.408716  1508 consensus_queue.cc:237] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.409089  1503 tablet_bootstrap.cc:492] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586: Bootstrap starting.
I20240503 08:20:50.412317  1503 tablet_bootstrap.cc:654] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:50.415925  1503 tablet_bootstrap.cc:492] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586: No bootstrap required, opened a new log
I20240503 08:20:50.415856   849 catalog_manager.cc:5484] T 7b1f90ec066648f6a289c2505354c04b P 4b12f470f64f4e29b3da57f18eee2586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "4b12f470f64f4e29b3da57f18eee2586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } } }
I20240503 08:20:50.416064  1503 ts_tablet_manager.cc:1367] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240503 08:20:50.416690  1503 raft_consensus.cc:348] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.416945  1503 raft_consensus.cc:374] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:50.417034  1503 raft_consensus.cc:729] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Initialized, Role: FOLLOWER
I20240503 08:20:50.417286  1503 consensus_queue.cc:260] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.417466  1503 raft_consensus.cc:388] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:50.417562  1503 raft_consensus.cc:482] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:50.417656  1503 raft_consensus.cc:3037] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:50.419972  1503 raft_consensus.cc:504] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.420248  1503 leader_election.cc:302] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b12f470f64f4e29b3da57f18eee2586; no voters: 
I20240503 08:20:50.420482  1503 leader_election.cc:288] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:50.420522  1508 raft_consensus.cc:2781] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:50.420686  1508 raft_consensus.cc:686] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 LEADER]: Becoming Leader. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Running, Role: LEADER
I20240503 08:20:50.420919  1508 consensus_queue.cc:237] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.421073  1503 ts_tablet_manager.cc:1398] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.001s
I20240503 08:20:50.423305   849 catalog_manager.cc:5484] T b8dfa0ed2ace4dc690f4292fca4a6369 P 4b12f470f64f4e29b3da57f18eee2586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "4b12f470f64f4e29b3da57f18eee2586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } } }
I20240503 08:20:50.945806   849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36422:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:20:50.956225   971 tablet_service.cc:1467] Processing CreateTablet for tablet 08f2959d15564f2ea809159b07838d76 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c24de883848746949e55642ae1ff4906]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:50.956382  1102 tablet_service.cc:1467] Processing CreateTablet for tablet daf26fdd3dd54ce09ea83a33b1630ee8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c24de883848746949e55642ae1ff4906]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:50.956499   970 tablet_service.cc:1467] Processing CreateTablet for tablet 794ddd6d72d646a5ad163341e322bbbc (DEFAULT_TABLE table=rebalance_test_table_2 [id=c24de883848746949e55642ae1ff4906]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:50.956755   971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f2959d15564f2ea809159b07838d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:50.956787  1102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf26fdd3dd54ce09ea83a33b1630ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:50.956966   970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794ddd6d72d646a5ad163341e322bbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:50.962757  1540 tablet_bootstrap.cc:492] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b: Bootstrap starting.
I20240503 08:20:50.963788  1541 tablet_bootstrap.cc:492] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586: Bootstrap starting.
I20240503 08:20:50.964718  1540 tablet_bootstrap.cc:654] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:50.965484  1541 tablet_bootstrap.cc:654] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:50.966754  1540 tablet_bootstrap.cc:492] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b: No bootstrap required, opened a new log
I20240503 08:20:50.966910  1540 ts_tablet_manager.cc:1367] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:20:50.967389  1541 tablet_bootstrap.cc:492] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586: No bootstrap required, opened a new log
I20240503 08:20:50.967543  1541 ts_tablet_manager.cc:1367] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:20:50.967715  1540 raft_consensus.cc:348] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.967904  1540 raft_consensus.cc:374] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:50.968003  1540 raft_consensus.cc:729] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Initialized, Role: FOLLOWER
I20240503 08:20:50.968250  1540 consensus_queue.cc:260] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.968310  1541 raft_consensus.cc:348] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.968473  1541 raft_consensus.cc:374] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:50.968474  1540 raft_consensus.cc:388] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:50.968559  1540 raft_consensus.cc:482] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:50.968570  1541 raft_consensus.cc:729] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Initialized, Role: FOLLOWER
I20240503 08:20:50.968688  1540 raft_consensus.cc:3037] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:50.968843  1541 consensus_queue.cc:260] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.969040  1541 raft_consensus.cc:388] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:50.969125  1541 raft_consensus.cc:482] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:50.969238  1541 raft_consensus.cc:3037] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:50.971541  1540 raft_consensus.cc:504] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.971840  1540 leader_election.cc:302] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed80600cd2b54ab08512e5301673272b; no voters: 
I20240503 08:20:50.972002  1541 raft_consensus.cc:504] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.972394  1540 leader_election.cc:288] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:50.972381  1541 leader_election.cc:302] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b12f470f64f4e29b3da57f18eee2586; no voters: 
I20240503 08:20:50.972508  1544 raft_consensus.cc:2781] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:50.972621  1541 leader_election.cc:288] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:50.972719  1508 raft_consensus.cc:2781] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:50.979815  1540 ts_tablet_manager.cc:1398] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.004s
I20240503 08:20:50.980275  1540 tablet_bootstrap.cc:492] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b: Bootstrap starting.
I20240503 08:20:50.982774  1540 tablet_bootstrap.cc:654] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:50.983052  1508 raft_consensus.cc:686] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 LEADER]: Becoming Leader. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Running, Role: LEADER
I20240503 08:20:50.983575  1508 consensus_queue.cc:237] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:50.989463   849 catalog_manager.cc:5484] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "4b12f470f64f4e29b3da57f18eee2586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } } }
I20240503 08:20:50.991788  1540 tablet_bootstrap.cc:492] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b: No bootstrap required, opened a new log
I20240503 08:20:50.992020  1540 ts_tablet_manager.cc:1367] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240503 08:20:50.993053  1540 raft_consensus.cc:348] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.993331  1540 raft_consensus.cc:374] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:50.993433  1540 raft_consensus.cc:729] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Initialized, Role: FOLLOWER
I20240503 08:20:50.993773  1540 consensus_queue.cc:260] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.994009  1540 raft_consensus.cc:388] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:50.994136  1540 raft_consensus.cc:482] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:50.994258  1540 raft_consensus.cc:3037] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:50.994805  1541 ts_tablet_manager.cc:1398] T daf26fdd3dd54ce09ea83a33b1630ee8 P 4b12f470f64f4e29b3da57f18eee2586: Time spent starting tablet: real 0.027s	user 0.019s	sys 0.009s
I20240503 08:20:50.994999  1544 raft_consensus.cc:686] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [term 1 LEADER]: Becoming Leader. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Running, Role: LEADER
I20240503 08:20:50.995301  1544 consensus_queue.cc:237] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.997097  1540 raft_consensus.cc:504] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:50.997419  1540 leader_election.cc:302] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed80600cd2b54ab08512e5301673272b; no voters: 
I20240503 08:20:50.997815  1545 raft_consensus.cc:2781] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:50.997896   849 catalog_manager.cc:5484] T 794ddd6d72d646a5ad163341e322bbbc P ed80600cd2b54ab08512e5301673272b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed80600cd2b54ab08512e5301673272b (127.2.49.129). New cstate: current_term: 1 leader_uuid: "ed80600cd2b54ab08512e5301673272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } } }
I20240503 08:20:51.003819  1540 leader_election.cc:288] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:51.004523  1540 ts_tablet_manager.cc:1398] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20240503 08:20:51.005028  1545 raft_consensus.cc:686] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [term 1 LEADER]: Becoming Leader. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Running, Role: LEADER
I20240503 08:20:51.005358  1545 consensus_queue.cc:237] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:51.009315   850 catalog_manager.cc:5484] T 08f2959d15564f2ea809159b07838d76 P ed80600cd2b54ab08512e5301673272b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed80600cd2b54ab08512e5301673272b (127.2.49.129). New cstate: current_term: 1 leader_uuid: "ed80600cd2b54ab08512e5301673272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } } }
I20240503 08:20:51.546435   850 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36422:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 08:20:51.554978  1102 tablet_service.cc:1467] Processing CreateTablet for tablet 4be40df76a874e3d936f2edcddacb135 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c29d623fa8d141bb965d979a918cdd11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:51.555133  1101 tablet_service.cc:1467] Processing CreateTablet for tablet 5caafbda0e3f461aaf99cd7f07d729aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=c29d623fa8d141bb965d979a918cdd11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:51.555517  1102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be40df76a874e3d936f2edcddacb135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:51.555591   971 tablet_service.cc:1467] Processing CreateTablet for tablet 6de6974cb29244daa6335c1a5378bbcf (DEFAULT_TABLE table=rebalance_test_table_3 [id=c29d623fa8d141bb965d979a918cdd11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:20:51.555801  1101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5caafbda0e3f461aaf99cd7f07d729aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:51.556026   971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de6974cb29244daa6335c1a5378bbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:51.562604  1579 tablet_bootstrap.cc:492] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586: Bootstrap starting.
I20240503 08:20:51.562873  1580 tablet_bootstrap.cc:492] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b: Bootstrap starting.
I20240503 08:20:51.564209  1580 tablet_bootstrap.cc:654] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:51.564216  1579 tablet_bootstrap.cc:654] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:51.566277  1580 tablet_bootstrap.cc:492] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b: No bootstrap required, opened a new log
I20240503 08:20:51.566381  1579 tablet_bootstrap.cc:492] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586: No bootstrap required, opened a new log
I20240503 08:20:51.566411  1580 ts_tablet_manager.cc:1367] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 08:20:51.566525  1579 ts_tablet_manager.cc:1367] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 08:20:51.567023  1580 raft_consensus.cc:348] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:51.567179  1580 raft_consensus.cc:374] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:51.567132  1579 raft_consensus.cc:348] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.567262  1580 raft_consensus.cc:729] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Initialized, Role: FOLLOWER
I20240503 08:20:51.567318  1579 raft_consensus.cc:374] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:51.567417  1579 raft_consensus.cc:729] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Initialized, Role: FOLLOWER
I20240503 08:20:51.567474  1580 consensus_queue.cc:260] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:51.567643  1580 raft_consensus.cc:388] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:51.567725  1580 raft_consensus.cc:482] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:51.567777  1579 consensus_queue.cc:260] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.567950  1579 raft_consensus.cc:388] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:51.567953  1580 raft_consensus.cc:3037] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:51.568050  1579 raft_consensus.cc:482] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:51.568167  1579 raft_consensus.cc:3037] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:51.570081  1580 raft_consensus.cc:504] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:51.570272  1579 raft_consensus.cc:504] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.570399  1580 leader_election.cc:302] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed80600cd2b54ab08512e5301673272b; no voters: 
I20240503 08:20:51.570531  1579 leader_election.cc:302] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b12f470f64f4e29b3da57f18eee2586; no voters: 
I20240503 08:20:51.570629  1580 leader_election.cc:288] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:51.570703  1545 raft_consensus.cc:2781] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:51.571045  1579 leader_election.cc:288] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:51.571267  1583 raft_consensus.cc:2781] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:51.577256  1583 raft_consensus.cc:686] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [term 1 LEADER]: Becoming Leader. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Running, Role: LEADER
I20240503 08:20:51.577859  1583 consensus_queue.cc:237] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.580691  1579 ts_tablet_manager.cc:1398] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.007s
I20240503 08:20:51.581094  1579 tablet_bootstrap.cc:492] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586: Bootstrap starting.
I20240503 08:20:51.582994  1579 tablet_bootstrap.cc:654] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:51.583361  1545 raft_consensus.cc:686] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [term 1 LEADER]: Becoming Leader. State: Replica: ed80600cd2b54ab08512e5301673272b, State: Running, Role: LEADER
I20240503 08:20:51.583719  1545 consensus_queue.cc:237] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } }
I20240503 08:20:51.584604  1580 ts_tablet_manager.cc:1398] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b: Time spent starting tablet: real 0.018s	user 0.013s	sys 0.004s
I20240503 08:20:51.586014   850 catalog_manager.cc:5484] T 4be40df76a874e3d936f2edcddacb135 P 4b12f470f64f4e29b3da57f18eee2586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "4b12f470f64f4e29b3da57f18eee2586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } } }
I20240503 08:20:51.589319  1579 tablet_bootstrap.cc:492] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586: No bootstrap required, opened a new log
I20240503 08:20:51.589488  1579 ts_tablet_manager.cc:1367] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240503 08:20:51.590266  1579 raft_consensus.cc:348] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.590458  1579 raft_consensus.cc:374] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:51.590528  1579 raft_consensus.cc:729] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Initialized, Role: FOLLOWER
I20240503 08:20:51.590798  1579 consensus_queue.cc:260] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.590961  1579 raft_consensus.cc:388] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:51.591042  1579 raft_consensus.cc:482] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:51.591128  1579 raft_consensus.cc:3037] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:51.594053  1579 raft_consensus.cc:504] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.594337  1579 leader_election.cc:302] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b12f470f64f4e29b3da57f18eee2586; no voters: 
I20240503 08:20:51.594592  1579 leader_election.cc:288] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:51.594874  1586 raft_consensus.cc:2781] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:51.595255  1586 raft_consensus.cc:686] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [term 1 LEADER]: Becoming Leader. State: Replica: 4b12f470f64f4e29b3da57f18eee2586, State: Running, Role: LEADER
I20240503 08:20:51.595052  1579 ts_tablet_manager.cc:1398] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20240503 08:20:51.595693  1586 consensus_queue.cc:237] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } }
I20240503 08:20:51.598605   849 catalog_manager.cc:5484] T 6de6974cb29244daa6335c1a5378bbcf P ed80600cd2b54ab08512e5301673272b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed80600cd2b54ab08512e5301673272b (127.2.49.129). New cstate: current_term: 1 leader_uuid: "ed80600cd2b54ab08512e5301673272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed80600cd2b54ab08512e5301673272b" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 43229 } } }
I20240503 08:20:51.604380   849 catalog_manager.cc:5484] T 5caafbda0e3f461aaf99cd7f07d729aa P 4b12f470f64f4e29b3da57f18eee2586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b12f470f64f4e29b3da57f18eee2586 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "4b12f470f64f4e29b3da57f18eee2586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b12f470f64f4e29b3da57f18eee2586" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 37085 } } }
I20240503 08:20:53.061548  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:38039
--local_ip_for_outbound_sockets=127.2.49.131
--tserver_master_addrs=127.2.49.190:37189
--webserver_port=44555
--webserver_interface=127.2.49.131
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:20:53.416869  1615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:53.417191  1615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:53.417285  1615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:53.425170  1615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:53.425308  1615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:53.425421  1615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:20:53.435001  1615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:38039
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=44555
--tserver_master_addrs=127.2.49.190:37189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:53.436331  1615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:53.438112  1615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:53.451395  1620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:53.451395  1621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:53.451908  1623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:53.452507  1615 server_base.cc:1034] running on GCE node
I20240503 08:20:53.453186  1615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:53.454298  1615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:53.455536  1615 hybrid_clock.cc:648] HybridClock initialized: now 1714724453455486 us; error 49 us; skew 500 ppm
I20240503 08:20:53.455920  1615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:53.458528  1615 webserver.cc:460] Webserver started at http://127.2.49.131:44555/ using document root <none> and password file <none>
I20240503 08:20:53.459323  1615 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:53.459462  1615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:53.465119  1615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.004s
I20240503 08:20:53.468204  1629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:53.469712  1615 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:20:53.469905  1615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "749f12d2084f43c5af05a839def4fe02"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:53.470914  1615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:53.507112  1615 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:53.507995  1615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:53.508335  1615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:53.509326  1615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:53.510947  1615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:53.511030  1615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:53.511171  1615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:53.511227  1615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:53.556032  1615 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:38039
I20240503 08:20:53.556134  1741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:38039 every 8 connection(s)
I20240503 08:20:53.557638  1615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:20:53.566787  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 1615
I20240503 08:20:53.567513  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:33473
--local_ip_for_outbound_sockets=127.2.49.132
--tserver_master_addrs=127.2.49.190:37189
--webserver_port=34979
--webserver_interface=127.2.49.132
--builtin_ntp_servers=127.2.49.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:20:53.575659  1742 heartbeater.cc:347] Connected to a master server at 127.2.49.190:37189
I20240503 08:20:53.576057  1742 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:53.577118  1742 heartbeater.cc:511] Master 127.2.49.190:37189 requested a full tablet report, sending...
I20240503 08:20:53.579154   849 ts_manager.cc:194] Re-registered known tserver with Master: 749f12d2084f43c5af05a839def4fe02 (127.2.49.131:38039)
I20240503 08:20:53.580395   849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:53739
W20240503 08:20:53.946697  1746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:53.947024  1746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:53.947171  1746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:53.955124  1746 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:20:53.955289  1746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:53.955453  1746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:20:53.965029  1746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:33473
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=34979
--tserver_master_addrs=127.2.49.190:37189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:53.966490  1746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:53.968283  1746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:53.982050  1754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:53.982090  1751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:53.982014  1752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:53.982762  1746 server_base.cc:1034] running on GCE node
I20240503 08:20:53.983592  1746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:53.984627  1746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:53.985849  1746 hybrid_clock.cc:648] HybridClock initialized: now 1714724453985807 us; error 47 us; skew 500 ppm
I20240503 08:20:53.986217  1746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:53.988677  1746 webserver.cc:460] Webserver started at http://127.2.49.132:34979/ using document root <none> and password file <none>
I20240503 08:20:53.989475  1746 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:53.989614  1746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:53.995147  1746 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 08:20:53.998283  1760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:53.999675  1746 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:20:53.999866  1746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "337e9dba83ca4221aa7dceb63124a5fe"
format_stamp: "Formatted at 2024-05-03 08:20:45 on dist-test-slave-dz87"
I20240503 08:20:54.000733  1746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:54.040813  1746 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:54.041725  1746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:54.042034  1746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:54.043094  1746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:54.045095  1767 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:20:54.050731  1746 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:20:54.050856  1746 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20240503 08:20:54.051009  1746 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:20:54.053893  1746 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:20:54.054003  1746 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
I20240503 08:20:54.054409  1767 tablet_bootstrap.cc:492] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Bootstrap starting.
I20240503 08:20:54.077347  1767 log.cc:824] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Log is configured to *not* fsync() on all Append() calls
I20240503 08:20:54.081362  1767 tablet_bootstrap.cc:492] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:20:54.081907  1767 tablet_bootstrap.cc:492] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Bootstrap complete.
I20240503 08:20:54.083483  1767 ts_tablet_manager.cc:1367] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.014s
I20240503 08:20:54.095234  1767 raft_consensus.cc:348] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:54.096364  1767 raft_consensus.cc:729] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337e9dba83ca4221aa7dceb63124a5fe, State: Initialized, Role: FOLLOWER
I20240503 08:20:54.097139  1767 consensus_queue.cc:260] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:54.097352  1767 raft_consensus.cc:388] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:54.097488  1767 raft_consensus.cc:482] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:54.097709  1767 raft_consensus.cc:3037] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:20:54.100958  1746 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:33473
I20240503 08:20:54.101047  1874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:33473 every 8 connection(s)
I20240503 08:20:54.102466  1767 raft_consensus.cc:504] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:54.102562  1746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:20:54.103014  1767 leader_election.cc:302] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 337e9dba83ca4221aa7dceb63124a5fe; no voters: 
I20240503 08:20:54.103713  1767 leader_election.cc:288] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [CANDIDATE]: Term 2 election: Requested vote from peers 
I20240503 08:20:54.103866  1876 raft_consensus.cc:2781] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:20:54.108212  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 1746
I20240503 08:20:54.121158  1876 raft_consensus.cc:686] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [term 2 LEADER]: Becoming Leader. State: Replica: 337e9dba83ca4221aa7dceb63124a5fe, State: Running, Role: LEADER
I20240503 08:20:54.121928  1876 consensus_queue.cc:237] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } }
I20240503 08:20:54.126053  1767 ts_tablet_manager.cc:1398] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe: Time spent starting tablet: real 0.042s	user 0.017s	sys 0.025s
I20240503 08:20:54.133116  1875 heartbeater.cc:347] Connected to a master server at 127.2.49.190:37189
I20240503 08:20:54.133443  1875 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:54.134310  1875 heartbeater.cc:511] Master 127.2.49.190:37189 requested a full tablet report, sending...
I20240503 08:20:54.136984   849 ts_manager.cc:194] Re-registered known tserver with Master: 337e9dba83ca4221aa7dceb63124a5fe (127.2.49.132:33473)
I20240503 08:20:54.137538   849 catalog_manager.cc:5484] T 6296a73ffe264abd952c6462a2d78e1f P 337e9dba83ca4221aa7dceb63124a5fe reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "337e9dba83ca4221aa7dceb63124a5fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337e9dba83ca4221aa7dceb63124a5fe" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 33473 } } }
I20240503 08:20:54.143646   849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:36269
I20240503 08:20:54.147931  1875 heartbeater.cc:503] Master 127.2.49.190:37189 was elected leader, sending a full tablet report...
I20240503 08:20:54.583621  1742 heartbeater.cc:503] Master 127.2.49.190:37189 was elected leader, sending a full tablet report...
I20240503 08:20:54.657946  1677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:54.671603  1809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:54.674118   971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:54.692081  1102 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:54.863164  1809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:54.863960  1677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:54.872563   971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:54.876035  1102 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:55.057475  1809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:55.070048  1677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:55.070426  1102 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:55.078752   971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:55.355953  1809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:20:55.365903  1102 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:20:55.377017  1677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:20:55.378304   971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 d274a4a7b9e44116abdebe167556b009 | 127.2.49.190:37189 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                               |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                              | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                 | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                             | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                              | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                               | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.2.49.148:37883 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 337e9dba83ca4221aa7dceb63124a5fe | 127.2.49.132:33473 | HEALTHY | <none>   |       1        |       0
 4b12f470f64f4e29b3da57f18eee2586 | 127.2.49.130:37085 | HEALTHY | <none>   |       6        |       0
 749f12d2084f43c5af05a839def4fe02 | 127.2.49.131:38039 | HEALTHY | <none>   |       0        |       0
 ed80600cd2b54ab08512e5301673272b | 127.2.49.129:43229 | HEALTHY | <none>   |       6        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                             |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                           | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.49.129                                                                                                                                                                                  | experimental          | 127.2.49.129:43229
 local_ip_for_outbound_sockets            | 127.2.49.130                                                                                                                                                                                  | experimental          | 127.2.49.130:37085
 local_ip_for_outbound_sockets            | 127.2.49.131                                                                                                                                                                                  | experimental          | 127.2.49.131:38039
 local_ip_for_outbound_sockets            | 127.2.49.132                                                                                                                                                                                  | experimental          | 127.2.49.132:33473
 never_fsync                              | true                                                                                                                                                                                          | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                             | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                         | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                          | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                            | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.49.129:43229
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.49.130:37085
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.49.131:38039
 server_dump_info_path                    | /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.49.132:33473

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.2.49.148:37883 | all 4 server(s) checked
 time_source         | builtin            | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 6
 Third Quartile | 6
 Maximum        | 6

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 08:20:55.434020  2246 log_verifier.cc:126] Checking tablet 08f2959d15564f2ea809159b07838d76
I20240503 08:20:55.442140  2246 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 08f2959d15564f2ea809159b07838d76
I20240503 08:20:55.442293  2246 log_verifier.cc:126] Checking tablet 0fa2cea1d8d04e16ac720bb7cd65c44f
I20240503 08:20:55.449947  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0fa2cea1d8d04e16ac720bb7cd65c44f
I20240503 08:20:55.450062  2246 log_verifier.cc:126] Checking tablet 298a0c656d0e4dca90fc5036126b1e71
I20240503 08:20:55.457724  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 298a0c656d0e4dca90fc5036126b1e71
I20240503 08:20:55.457816  2246 log_verifier.cc:126] Checking tablet 4be40df76a874e3d936f2edcddacb135
I20240503 08:20:55.465654  2246 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 4be40df76a874e3d936f2edcddacb135
I20240503 08:20:55.465747  2246 log_verifier.cc:126] Checking tablet 5caafbda0e3f461aaf99cd7f07d729aa
I20240503 08:20:55.473165  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5caafbda0e3f461aaf99cd7f07d729aa
I20240503 08:20:55.473268  2246 log_verifier.cc:126] Checking tablet 6296a73ffe264abd952c6462a2d78e1f
I20240503 08:20:55.480841  2246 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6296a73ffe264abd952c6462a2d78e1f
I20240503 08:20:55.480949  2246 log_verifier.cc:126] Checking tablet 63c88dbb9b40439795b7c189b4095d63
I20240503 08:20:55.489157  2246 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 63c88dbb9b40439795b7c189b4095d63
I20240503 08:20:55.489256  2246 log_verifier.cc:126] Checking tablet 6de6974cb29244daa6335c1a5378bbcf
I20240503 08:20:55.496917  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6de6974cb29244daa6335c1a5378bbcf
I20240503 08:20:55.497013  2246 log_verifier.cc:126] Checking tablet 794ddd6d72d646a5ad163341e322bbbc
I20240503 08:20:55.504762  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 794ddd6d72d646a5ad163341e322bbbc
I20240503 08:20:55.504851  2246 log_verifier.cc:126] Checking tablet 7b1f90ec066648f6a289c2505354c04b
I20240503 08:20:55.512861  2246 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 7b1f90ec066648f6a289c2505354c04b
I20240503 08:20:55.512953  2246 log_verifier.cc:126] Checking tablet b8dfa0ed2ace4dc690f4292fca4a6369
I20240503 08:20:55.562093  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b8dfa0ed2ace4dc690f4292fca4a6369
I20240503 08:20:55.562196  2246 log_verifier.cc:126] Checking tablet d9ecca63cc22418abc3bd2a6e7f2bce8
I20240503 08:20:55.569705  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d9ecca63cc22418abc3bd2a6e7f2bce8
I20240503 08:20:55.569787  2246 log_verifier.cc:126] Checking tablet daf26fdd3dd54ce09ea83a33b1630ee8
I20240503 08:20:55.577260  2246 log_verifier.cc:177] Verified matching terms for 1 ops in tablet daf26fdd3dd54ce09ea83a33b1630ee8
I20240503 08:20:55.578186  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 909
I20240503 08:20:55.579392  1030 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:55.684407  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 909
W20240503 08:20:55.711977   835 connection.cc:535] server connection from 127.2.49.129:42807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:20:55.712296  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 1040
I20240503 08:20:55.713339  1162 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:55.814788  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 1040
I20240503 08:20:55.836305  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 1615
I20240503 08:20:55.842564  1737 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:55.934139  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 1615
I20240503 08:20:55.953009  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 1746
I20240503 08:20:55.959113  1870 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:56.054113  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 1746
I20240503 08:20:56.073734  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 818
I20240503 08:20:56.074867   880 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:56.197934  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 818
2024-05-03T08:20:56Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (13264 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (13264 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240503 08:20:56.259088  2246 test_util.cc:268] Using random seed: 1713004014
I20240503 08:20:56.261018  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:20:56.261091  2246 ts_itest-base.cc:113] --------------
I20240503 08:20:56.261126  2246 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:20:56.261157  2246 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:20:56.261191  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:20:56Z Disabled control of system clock
I20240503 08:20:56.318538  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:43231
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:43231
--location_mapping_cmd=/tmp/dist-test-taskPaEuZ9/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:20:56.673743  2039 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:20:56.674111  2039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:56.674229  2039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:56.674314  2039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:56.681995  2039 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:20:56.682129  2039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:20:56.682251  2039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:56.682304  2039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:20:56.682368  2039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:20:56.691874  2039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskPaEuZ9/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:43231
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:56.693202  2039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:56.695008  2039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:56.705683  2044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:56.705713  2047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:56.706346  2045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:56.706560  2039 server_base.cc:1034] running on GCE node
I20240503 08:20:56.707362  2039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:56.708842  2039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:56.710075  2039 hybrid_clock.cc:648] HybridClock initialized: now 1714724456710012 us; error 38 us; skew 500 ppm
I20240503 08:20:56.710420  2039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:56.712798  2039 webserver.cc:460] Webserver started at http://127.2.49.190:34891/ using document root <none> and password file <none>
I20240503 08:20:56.713528  2039 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:56.713670  2039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:56.713970  2039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:56.716807  2039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3a378405f3ea4f489a9ffc7fb4d50794"
format_stamp: "Formatted at 2024-05-03 08:20:56 on dist-test-slave-dz87"
I20240503 08:20:56.717875  2039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3a378405f3ea4f489a9ffc7fb4d50794"
format_stamp: "Formatted at 2024-05-03 08:20:56 on dist-test-slave-dz87"
I20240503 08:20:56.723651  2039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:20:56.727228  2053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:56.728624  2039 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:20:56.728827  2039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3a378405f3ea4f489a9ffc7fb4d50794"
format_stamp: "Formatted at 2024-05-03 08:20:56 on dist-test-slave-dz87"
I20240503 08:20:56.729023  2039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:56.776180  2039 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:56.777086  2039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:56.777395  2039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:56.795509  2039 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:43231
I20240503 08:20:56.795567  2104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:43231 every 8 connection(s)
I20240503 08:20:56.796960  2039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:20:56.800789  2105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:20:56.803977  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2039
I20240503 08:20:56.804214  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:20:56.813031  2105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794: Bootstrap starting.
I20240503 08:20:56.816489  2105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794: Neither blocks nor log segments found. Creating new log.
I20240503 08:20:56.817891  2105 log.cc:824] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794: Log is configured to *not* fsync() on all Append() calls
I20240503 08:20:56.820667  2105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794: No bootstrap required, opened a new log
I20240503 08:20:56.827368  2105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 43231 } }
I20240503 08:20:56.827836  2105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:20:56.827955  2105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a378405f3ea4f489a9ffc7fb4d50794, State: Initialized, Role: FOLLOWER
I20240503 08:20:56.828631  2105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 43231 } }
I20240503 08:20:56.828852  2105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:20:56.828979  2105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:20:56.829135  2105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:20:56.831542  2105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 43231 } }
I20240503 08:20:56.832072  2105 leader_election.cc:302] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a378405f3ea4f489a9ffc7fb4d50794; no voters: 
I20240503 08:20:56.832711  2105 leader_election.cc:288] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:20:56.832849  2110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:20:56.833545  2110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [term 1 LEADER]: Becoming Leader. State: Replica: 3a378405f3ea4f489a9ffc7fb4d50794, State: Running, Role: LEADER
I20240503 08:20:56.834200  2110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 43231 } }
I20240503 08:20:56.834985  2105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:20:56.837518  2111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 43231 } } }
I20240503 08:20:56.837955  2111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [sys.catalog]: This master's current role is: LEADER
I20240503 08:20:56.839849  2112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a378405f3ea4f489a9ffc7fb4d50794. Latest consensus state: current_term: 1 leader_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a378405f3ea4f489a9ffc7fb4d50794" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 43231 } } }
I20240503 08:20:56.840282  2112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794 [sys.catalog]: This master's current role is: LEADER
I20240503 08:20:56.840714  2118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:20:56.846220  2118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:20:56.855866  2118 catalog_manager.cc:1347] Generated new cluster ID: d854af20704f4ffab743dfe4ab45fbdf
I20240503 08:20:56.855994  2118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:20:56.868301  2118 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:20:56.869558  2118 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:20:56.882066  2118 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3a378405f3ea4f489a9ffc7fb4d50794: Generated new TSK 0
I20240503 08:20:56.883189  2118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:20:56.895134  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:20:57.260857  2129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:57.261178  2129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:57.261329  2129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:57.269026  2129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:20:57.269163  2129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:57.269331  2129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:20:57.278818  2129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:57.280185  2129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:57.282037  2129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:57.294234  2135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:57.294337  2134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:57.295051  2137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:57.295068  2129 server_base.cc:1034] running on GCE node
I20240503 08:20:57.295734  2129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:57.296718  2129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:57.297925  2129 hybrid_clock.cc:648] HybridClock initialized: now 1714724457297878 us; error 47 us; skew 500 ppm
I20240503 08:20:57.298274  2129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:57.300673  2129 webserver.cc:460] Webserver started at http://127.2.49.129:42505/ using document root <none> and password file <none>
I20240503 08:20:57.301410  2129 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:57.301532  2129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:57.301841  2129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:57.304556  2129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "03bd4573b6f64823bb1b514c106baed6"
format_stamp: "Formatted at 2024-05-03 08:20:57 on dist-test-slave-dz87"
I20240503 08:20:57.305346  2129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "03bd4573b6f64823bb1b514c106baed6"
format_stamp: "Formatted at 2024-05-03 08:20:57 on dist-test-slave-dz87"
I20240503 08:20:57.310884  2129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240503 08:20:57.314620  2143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:57.316073  2129 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:20:57.316249  2129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "03bd4573b6f64823bb1b514c106baed6"
format_stamp: "Formatted at 2024-05-03 08:20:57 on dist-test-slave-dz87"
I20240503 08:20:57.316473  2129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:57.331665  2129 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:57.332579  2129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:57.332885  2129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:57.333899  2129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:57.335523  2129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:57.335621  2129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:57.335738  2129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:57.335808  2129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:57.368249  2129 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:40211
I20240503 08:20:57.368335  2261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:40211 every 8 connection(s)
I20240503 08:20:57.369607  2129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:20:57.370635  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2129
I20240503 08:20:57.370853  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:20:57.374743  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:20:57.388969  2262 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:20:57.389379  2262 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:57.390408  2262 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:20:57.460959  2070 ts_manager.cc:194] Registered new tserver with Master: 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:20:57.462860  2070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:51681
W20240503 08:20:57.757210  2266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:57.757560  2266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:57.757710  2266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:57.765585  2266 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:20:57.765723  2266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:57.765890  2266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:20:57.775256  2266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:57.776641  2266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:57.778455  2266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:57.790935  2272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:57.790989  2275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:57.790880  2273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:57.791869  2266 server_base.cc:1034] running on GCE node
I20240503 08:20:57.792507  2266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:57.793506  2266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:57.794704  2266 hybrid_clock.cc:648] HybridClock initialized: now 1714724457794657 us; error 38 us; skew 500 ppm
I20240503 08:20:57.795068  2266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:57.797549  2266 webserver.cc:460] Webserver started at http://127.2.49.130:45421/ using document root <none> and password file <none>
I20240503 08:20:57.798329  2266 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:57.798450  2266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:57.798775  2266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:57.801426  2266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2b568c777ade427b9af3dae3395eaaa2"
format_stamp: "Formatted at 2024-05-03 08:20:57 on dist-test-slave-dz87"
I20240503 08:20:57.802343  2266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2b568c777ade427b9af3dae3395eaaa2"
format_stamp: "Formatted at 2024-05-03 08:20:57 on dist-test-slave-dz87"
I20240503 08:20:57.807999  2266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:20:57.811580  2281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:57.812928  2266 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:20:57.813097  2266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2b568c777ade427b9af3dae3395eaaa2"
format_stamp: "Formatted at 2024-05-03 08:20:57 on dist-test-slave-dz87"
I20240503 08:20:57.813304  2266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:57.836548  2266 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:57.837462  2266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:57.837729  2266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:57.838706  2266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:57.840226  2266 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:57.840314  2266 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:57.840394  2266 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:57.840448  2266 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:57.871215  2266 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:34357
I20240503 08:20:57.871263  2393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:34357 every 8 connection(s)
I20240503 08:20:57.872582  2266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:20:57.877816  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2266
I20240503 08:20:57.877976  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:20:57.881420  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:20:57.885922  2394 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:20:57.886302  2394 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:57.887367  2394 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:20:57.934581  2070 ts_manager.cc:194] Registered new tserver with Master: 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:20:57.935864  2070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:50967
W20240503 08:20:58.248335  2397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:58.248653  2397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:58.248754  2397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:58.256434  2397 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:20:58.256573  2397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:58.256698  2397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:20:58.266206  2397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:58.267617  2397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:58.269333  2397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:58.281500  2406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:58.282130  2404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:58.282363  2403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:58.282886  2397 server_base.cc:1034] running on GCE node
I20240503 08:20:58.283519  2397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:58.284477  2397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:58.285717  2397 hybrid_clock.cc:648] HybridClock initialized: now 1714724458285670 us; error 44 us; skew 500 ppm
I20240503 08:20:58.286049  2397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:58.288452  2397 webserver.cc:460] Webserver started at http://127.2.49.131:43915/ using document root <none> and password file <none>
I20240503 08:20:58.289222  2397 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:58.289350  2397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:58.289673  2397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:58.292426  2397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:20:58.293264  2397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:20:58.298715  2397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 08:20:58.302413  2412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:58.303826  2397 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:20:58.303998  2397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:20:58.304211  2397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:58.319319  2397 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:58.320204  2397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:58.320502  2397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:58.321528  2397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:58.323292  2397 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:58.323395  2397 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:58.323508  2397 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:58.323585  2397 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:58.357882  2397 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:37311
I20240503 08:20:58.358017  2524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:37311 every 8 connection(s)
I20240503 08:20:58.359329  2397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:20:58.368505  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2397
I20240503 08:20:58.368729  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:20:58.372494  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:20:58.373440  2525 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:20:58.373823  2525 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:58.374913  2525 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:20:58.444028  2070 ts_manager.cc:194] Registered new tserver with Master: b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:20:58.445036  2070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:41461
I20240503 08:20:58.466301  2262 heartbeater.cc:503] Master 127.2.49.190:43231 was elected leader, sending a full tablet report...
W20240503 08:20:58.749935  2529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:58.750303  2529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:58.750468  2529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:58.758039  2529 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:20:58.758230  2529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:58.758402  2529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:20:58.767592  2529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:58.768956  2529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:58.770779  2529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:58.782845  2534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:58.782876  2535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:58.783476  2537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:58.783994  2529 server_base.cc:1034] running on GCE node
I20240503 08:20:58.784626  2529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:58.785673  2529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:58.786873  2529 hybrid_clock.cc:648] HybridClock initialized: now 1714724458786810 us; error 64 us; skew 500 ppm
I20240503 08:20:58.787235  2529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:58.789619  2529 webserver.cc:460] Webserver started at http://127.2.49.132:35399/ using document root <none> and password file <none>
I20240503 08:20:58.790407  2529 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:58.790520  2529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:58.790803  2529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:58.793493  2529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ca50a57f43c9465ba7d4049fd0728361"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:20:58.794348  2529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ca50a57f43c9465ba7d4049fd0728361"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:20:58.799901  2529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:20:58.803579  2543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:58.804980  2529 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:20:58.805157  2529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ca50a57f43c9465ba7d4049fd0728361"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:20:58.805369  2529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:58.832756  2529 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:58.833822  2529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:58.834123  2529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:58.835366  2529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:58.837007  2529 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:58.837101  2529 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:58.837185  2529 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:58.837240  2529 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:58.871452  2529 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:34267
I20240503 08:20:58.871546  2655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:34267 every 8 connection(s)
I20240503 08:20:58.872992  2529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:20:58.879665  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2529
I20240503 08:20:58.879864  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:20:58.883342  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:0
--local_ip_for_outbound_sockets=127.2.49.133
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:20:58.887540  2656 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:20:58.887888  2656 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:58.888794  2656 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:20:58.936537  2070 ts_manager.cc:194] Registered new tserver with Master: ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132:34267)
I20240503 08:20:58.937880  2070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:52591
I20240503 08:20:58.939199  2394 heartbeater.cc:503] Master 127.2.49.190:43231 was elected leader, sending a full tablet report...
W20240503 08:20:59.260891  2660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:20:59.261229  2660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:20:59.261384  2660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:20:59.268941  2660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:20:59.269101  2660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:20:59.269275  2660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:20:59.278492  2660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=0
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:20:59.279844  2660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:20:59.281580  2660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:20:59.293987  2667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:20:59.294231  2666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:59.294701  2660 server_base.cc:1034] running on GCE node
W20240503 08:20:59.295650  2669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:20:59.296308  2660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:20:59.297289  2660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:20:59.298511  2660 hybrid_clock.cc:648] HybridClock initialized: now 1714724459298434 us; error 54 us; skew 500 ppm
I20240503 08:20:59.298842  2660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:20:59.301198  2660 webserver.cc:460] Webserver started at http://127.2.49.133:46669/ using document root <none> and password file <none>
I20240503 08:20:59.301970  2660 fs_manager.cc:359] Metadata directory not provided
I20240503 08:20:59.302099  2660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:20:59.302438  2660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:20:59.305116  2660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "aef8f8b349c44f91ad2c723851a827a0"
format_stamp: "Formatted at 2024-05-03 08:20:59 on dist-test-slave-dz87"
I20240503 08:20:59.305933  2660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "aef8f8b349c44f91ad2c723851a827a0"
format_stamp: "Formatted at 2024-05-03 08:20:59 on dist-test-slave-dz87"
I20240503 08:20:59.311947  2660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.000s
I20240503 08:20:59.315873  2675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:59.317404  2660 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:20:59.317579  2660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "aef8f8b349c44f91ad2c723851a827a0"
format_stamp: "Formatted at 2024-05-03 08:20:59 on dist-test-slave-dz87"
I20240503 08:20:59.317823  2660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:20:59.371251  2660 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:20:59.372174  2660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:20:59.372462  2660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:20:59.373485  2660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:20:59.375224  2660 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:20:59.375334  2660 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:59.375451  2660 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:20:59.375523  2660 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:20:59.409444  2660 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:38143
I20240503 08:20:59.409523  2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:38143 every 8 connection(s)
I20240503 08:20:59.410923  2660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:20:59.420898  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 2660
I20240503 08:20:59.421116  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:20:59.424387  2788 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:20:59.424760  2788 heartbeater.cc:464] Registering TS with master...
I20240503 08:20:59.425724  2788 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:20:59.448619  2525 heartbeater.cc:503] Master 127.2.49.190:43231 was elected leader, sending a full tablet report...
I20240503 08:20:59.473006  2070 ts_manager.cc:194] Registered new tserver with Master: aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143)
I20240503 08:20:59.473968  2070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:39033
I20240503 08:20:59.476354  2246 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:20:59.492914  2246 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:20:59.493067  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2529
I20240503 08:20:59.500788  2651 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:59.581993  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2529
I20240503 08:20:59.599507  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2660
I20240503 08:20:59.608141  2783 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:20:59.689401  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2660
I20240503 08:21:03.458976  2070 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56190:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 08:21:03.465866  2070 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:21:03.505416  2191 tablet_service.cc:1467] Processing CreateTablet for tablet 6836fefc21b64f9fadcdcd05794fe7bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.505481  2459 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe0304979f4d668f9b44b954783e17 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.505437  2460 tablet_service.cc:1467] Processing CreateTablet for tablet 6836fefc21b64f9fadcdcd05794fe7bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.506067  2329 tablet_service.cc:1467] Processing CreateTablet for tablet 6836fefc21b64f9fadcdcd05794fe7bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.507141  2459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe0304979f4d668f9b44b954783e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.507642  2458 tablet_service.cc:1467] Processing CreateTablet for tablet 4510323bcd68498287476b09df0e8a2e (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.507797  2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836fefc21b64f9fadcdcd05794fe7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.507959  2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836fefc21b64f9fadcdcd05794fe7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.508193  2458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4510323bcd68498287476b09df0e8a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.506035  2190 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe0304979f4d668f9b44b954783e17 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.506985  2189 tablet_service.cc:1467] Processing CreateTablet for tablet 4510323bcd68498287476b09df0e8a2e (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.510208  2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836fefc21b64f9fadcdcd05794fe7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.510673  2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe0304979f4d668f9b44b954783e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.510892  2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4510323bcd68498287476b09df0e8a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.517966  2328 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe0304979f4d668f9b44b954783e17 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.518540  2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe0304979f4d668f9b44b954783e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.526576  2327 tablet_service.cc:1467] Processing CreateTablet for tablet 4510323bcd68498287476b09df0e8a2e (DEFAULT_TABLE table=rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:03.527138  2327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4510323bcd68498287476b09df0e8a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:03.532164  2813 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:03.537006  2813 tablet_bootstrap.cc:654] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.537786  2815 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:03.538467  2813 log.cc:824] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:03.539304  2814 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:03.541481  2815 tablet_bootstrap.cc:654] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.541749  2813 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:03.542068  2813 ts_tablet_manager.cc:1367] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240503 08:21:03.542204  2814 tablet_bootstrap.cc:654] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.543246  2814 log.cc:824] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:03.543449  2815 log.cc:824] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:03.546820  2814 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:03.547109  2814 ts_tablet_manager.cc:1367] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240503 08:21:03.548244  2815 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:03.548552  2815 ts_tablet_manager.cc:1367] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.000s
I20240503 08:21:03.551999  2813 raft_consensus.cc:348] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.552511  2813 raft_consensus.cc:374] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.552683  2813 raft_consensus.cc:729] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.553506  2813 consensus_queue.cc:260] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.554824  2814 raft_consensus.cc:348] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.555189  2814 raft_consensus.cc:374] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.555295  2814 raft_consensus.cc:729] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.555904  2814 consensus_queue.cc:260] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.556871  2813 ts_tablet_manager.cc:1398] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.006s
I20240503 08:21:03.557379  2813 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:03.557466  2815 raft_consensus.cc:348] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.557929  2815 raft_consensus.cc:374] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.558054  2815 raft_consensus.cc:729] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.558969  2815 consensus_queue.cc:260] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.559334  2813 tablet_bootstrap.cc:654] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.567356  2814 ts_tablet_manager.cc:1398] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.007s
I20240503 08:21:03.567466  2815 ts_tablet_manager.cc:1398] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.019s	user 0.011s	sys 0.005s
I20240503 08:21:03.568158  2815 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:03.569315  2814 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:03.570674  2815 tablet_bootstrap.cc:654] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.572989  2813 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:03.573230  2813 ts_tablet_manager.cc:1367] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.009s
I20240503 08:21:03.573585  2814 tablet_bootstrap.cc:654] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.573700  2815 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:03.573860  2815 ts_tablet_manager.cc:1367] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 08:21:03.574438  2813 raft_consensus.cc:348] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.574673  2813 raft_consensus.cc:374] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.574816  2813 raft_consensus.cc:729] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.574864  2815 raft_consensus.cc:348] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.575126  2815 raft_consensus.cc:374] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.575246  2815 raft_consensus.cc:729] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.575152  2813 consensus_queue.cc:260] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.575479  2814 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:03.575471  2815 consensus_queue.cc:260] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.575670  2814 ts_tablet_manager.cc:1367] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240503 08:21:03.575724  2813 ts_tablet_manager.cc:1398] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:03.576035  2815 ts_tablet_manager.cc:1398] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:03.576086  2813 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:03.576354  2815 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:03.577804  2813 tablet_bootstrap.cc:654] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.577934  2815 tablet_bootstrap.cc:654] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.576792  2814 raft_consensus.cc:348] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.578265  2814 raft_consensus.cc:374] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.578418  2814 raft_consensus.cc:729] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.578716  2814 consensus_queue.cc:260] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.579938  2814 ts_tablet_manager.cc:1398] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:21:03.580261  2814 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:03.582037  2814 tablet_bootstrap.cc:654] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:03.585829  2815 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:03.586076  2815 ts_tablet_manager.cc:1367] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.000s
I20240503 08:21:03.587117  2815 raft_consensus.cc:348] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.587474  2815 raft_consensus.cc:374] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.587577  2815 raft_consensus.cc:729] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.587914  2815 consensus_queue.cc:260] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.588735  2815 ts_tablet_manager.cc:1398] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.000s
I20240503 08:21:03.590983  2813 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:03.591075  2814 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:03.591257  2813 ts_tablet_manager.cc:1367] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.008s
I20240503 08:21:03.591259  2814 ts_tablet_manager.cc:1367] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.006s
I20240503 08:21:03.592015  2813 raft_consensus.cc:348] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.592015  2814 raft_consensus.cc:348] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.592267  2813 raft_consensus.cc:374] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.592267  2814 raft_consensus.cc:374] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:03.592350  2813 raft_consensus.cc:729] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.592350  2814 raft_consensus.cc:729] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:03.592624  2814 consensus_queue.cc:260] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.592665  2813 consensus_queue.cc:260] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.593364  2813 ts_tablet_manager.cc:1398] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:21:03.593771  2814 ts_tablet_manager.cc:1398] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
W20240503 08:21:03.614898  2526 tablet.cc:2369] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:21:03.626185  2263 tablet.cc:2369] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:21:03.628368  2395 tablet.cc:2369] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:21:03.652487  2820 raft_consensus.cc:482] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:03.652817  2820 raft_consensus.cc:504] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.655230  2820 leader_election.cc:288] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311), 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:03.663743  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:03.663913  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:03.664281  2480 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:03.664381  2211 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:03.664902  2282 leader_election.cc:302] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:03.665349  2820 raft_consensus.cc:2781] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:03.665493  2820 raft_consensus.cc:482] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:03.665601  2820 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.667709  2820 raft_consensus.cc:504] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.668304  2820 leader_election.cc:288] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Requested vote from peers b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311), 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:03.668777  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:03.668807  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:03.668964  2480 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.668980  2211 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.671113  2480 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:03.671118  2211 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:03.671566  2284 leader_election.cc:302] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:03.671875  2820 raft_consensus.cc:2781] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:03.672281  2820 raft_consensus.cc:686] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:03.672729  2820 consensus_queue.cc:237] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } }
I20240503 08:21:03.677511  2070 catalog_manager.cc:5484] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:03.741153  2820 raft_consensus.cc:482] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:03.741475  2820 raft_consensus.cc:504] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.742259  2820 leader_election.cc:288] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:03.742883  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe0304979f4d668f9b44b954783e17" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:03.742926  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe0304979f4d668f9b44b954783e17" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:03.743144  2211 raft_consensus.cc:2454] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:03.743181  2480 raft_consensus.cc:2454] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:03.743537  2284 leader_election.cc:302] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:03.743885  2820 raft_consensus.cc:2781] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:03.743981  2820 raft_consensus.cc:482] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:03.744093  2820 raft_consensus.cc:3037] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.746096  2820 raft_consensus.cc:504] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.746773  2820 leader_election.cc:288] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Requested vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:03.747306  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe0304979f4d668f9b44b954783e17" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:03.747305  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe0304979f4d668f9b44b954783e17" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:03.747570  2211 raft_consensus.cc:3037] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.747582  2480 raft_consensus.cc:3037] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.750545  2211 raft_consensus.cc:2454] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:03.750543  2480 raft_consensus.cc:2454] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:03.751065  2284 leader_election.cc:302] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:03.751396  2820 raft_consensus.cc:2781] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:03.751595  2820 raft_consensus.cc:686] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:03.751859  2820 consensus_queue.cc:237] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.754487  2070 catalog_manager.cc:5484] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:03.840829  2821 raft_consensus.cc:482] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:03.841071  2821 raft_consensus.cc:504] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.842797  2821 leader_election.cc:288] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:03.849645  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4510323bcd68498287476b09df0e8a2e" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:03.849936  2480 raft_consensus.cc:2454] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 0.
I20240503 08:21:03.849856  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4510323bcd68498287476b09df0e8a2e" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2" is_pre_election: true
I20240503 08:21:03.850323  2349 raft_consensus.cc:2454] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 0.
I20240503 08:21:03.850473  2144 leader_election.cc:302] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:03.850899  2821 raft_consensus.cc:2781] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:03.851054  2821 raft_consensus.cc:482] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:03.851307  2821 raft_consensus.cc:3037] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.853338  2821 raft_consensus.cc:504] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.853946  2821 leader_election.cc:288] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Requested vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:03.854431  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4510323bcd68498287476b09df0e8a2e" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
I20240503 08:21:03.854449  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4510323bcd68498287476b09df0e8a2e" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:03.854590  2349 raft_consensus.cc:3037] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.854627  2480 raft_consensus.cc:3037] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:03.856448  2480 raft_consensus.cc:2454] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 1.
I20240503 08:21:03.856541  2349 raft_consensus.cc:2454] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 1.
I20240503 08:21:03.856868  2144 leader_election.cc:302] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:03.857203  2821 raft_consensus.cc:2781] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:03.857630  2821 raft_consensus.cc:686] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Becoming Leader. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Running, Role: LEADER
I20240503 08:21:03.858203  2821 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:03.862780  2070 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:04.053925  2820 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:04.060688  2820 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:04.170217  2820 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:04.176712  2844 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:04.297767  2837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:04.298103  2837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:04.305691  2837 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:21:04.375216  2821 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:04.451740  2821 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 288.64 ms
  time per row: 0.14432 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 08:21:05.682396  2064 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56190:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 08:21:05.683271  2064 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:21:05.692135  2189 tablet_service.cc:1467] Processing CreateTablet for tablet cfc12dd5961041318d235c7e581238d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.692232  2328 tablet_service.cc:1467] Processing CreateTablet for tablet cfc12dd5961041318d235c7e581238d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.692621  2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc12dd5961041318d235c7e581238d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.692638  2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc12dd5961041318d235c7e581238d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.693179  2329 tablet_service.cc:1467] Processing CreateTablet for tablet 361d2edf4fbb4781b23245a436b49f9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.693568  2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361d2edf4fbb4781b23245a436b49f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.693778  2460 tablet_service.cc:1467] Processing CreateTablet for tablet 361d2edf4fbb4781b23245a436b49f9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.694195  2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361d2edf4fbb4781b23245a436b49f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.694619  2459 tablet_service.cc:1467] Processing CreateTablet for tablet cfc12dd5961041318d235c7e581238d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.695019  2459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc12dd5961041318d235c7e581238d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.698774  2191 tablet_service.cc:1467] Processing CreateTablet for tablet 361d2edf4fbb4781b23245a436b49f9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.699211  2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361d2edf4fbb4781b23245a436b49f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.701081  2458 tablet_service.cc:1467] Processing CreateTablet for tablet e21365f867974e38bef15b340748d8ae (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.701601  2458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21365f867974e38bef15b340748d8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.704059  2327 tablet_service.cc:1467] Processing CreateTablet for tablet e21365f867974e38bef15b340748d8ae (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.704638  2327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21365f867974e38bef15b340748d8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.705798  2900 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:05.707648  2900 tablet_bootstrap.cc:654] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.708554  2190 tablet_service.cc:1467] Processing CreateTablet for tablet e21365f867974e38bef15b340748d8ae (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:05.708982  2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21365f867974e38bef15b340748d8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:05.715332  2901 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:05.716509  2902 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:05.717339  2901 tablet_bootstrap.cc:654] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.719189  2902 tablet_bootstrap.cc:654] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.721987  2900 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:05.722151  2900 ts_tablet_manager.cc:1367] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.012s
I20240503 08:21:05.722316  2901 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:05.722476  2901 ts_tablet_manager.cc:1367] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 08:21:05.722887  2902 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:05.723047  2902 ts_tablet_manager.cc:1367] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.005s
I20240503 08:21:05.723022  2900 raft_consensus.cc:348] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.723177  2901 raft_consensus.cc:348] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.723343  2900 raft_consensus.cc:374] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.723443  2901 raft_consensus.cc:374] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.723491  2900 raft_consensus.cc:729] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.723642  2901 raft_consensus.cc:729] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.723773  2900 consensus_queue.cc:260] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.723870  2902 raft_consensus.cc:348] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.724113  2902 raft_consensus.cc:374] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.723963  2901 consensus_queue.cc:260] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.724253  2902 raft_consensus.cc:729] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.724537  2902 consensus_queue.cc:260] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.724838  2871 raft_consensus.cc:482] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:05.724968  2871 raft_consensus.cc:504] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.725503  2900 ts_tablet_manager.cc:1398] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:21:05.725675  2871 leader_election.cc:288] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:05.725894  2901 ts_tablet_manager.cc:1398] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:21:05.726198  2901 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:05.726445  2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361d2edf4fbb4781b23245a436b49f9b" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:05.726524  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361d2edf4fbb4781b23245a436b49f9b" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2" is_pre_election: true
I20240503 08:21:05.726729  2349 raft_consensus.cc:2454] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 0.
I20240503 08:21:05.727296  2146 leader_election.cc:302] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
W20240503 08:21:05.727494  2144 leader_election.cc:341] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:21:05.727921  2901 tablet_bootstrap.cc:654] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.729223  2900 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:05.730810  2871 raft_consensus.cc:2781] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:05.730980  2871 raft_consensus.cc:482] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:05.731128  2871 raft_consensus.cc:3037] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.731810  2900 tablet_bootstrap.cc:654] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.734898  2902 ts_tablet_manager.cc:1398] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.008s
I20240503 08:21:05.735671  2902 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:05.738296  2902 tablet_bootstrap.cc:654] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.739169  2871 raft_consensus.cc:504] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.740134  2900 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:05.740346  2900 ts_tablet_manager.cc:1367] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.009s
I20240503 08:21:05.741036  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361d2edf4fbb4781b23245a436b49f9b" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
I20240503 08:21:05.741317  2349 raft_consensus.cc:3037] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.743108  2902 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:05.743283  2902 ts_tablet_manager.cc:1367] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240503 08:21:05.744122  2902 raft_consensus.cc:348] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.744239  2349 raft_consensus.cc:2454] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 1.
I20240503 08:21:05.744361  2902 raft_consensus.cc:374] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.744453  2902 raft_consensus.cc:729] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.744756  2146 leader_election.cc:302] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:05.745283  2900 raft_consensus.cc:348] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.745523  2900 raft_consensus.cc:374] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.745595  2900 raft_consensus.cc:729] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.745858  2900 consensus_queue.cc:260] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.745869  2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361d2edf4fbb4781b23245a436b49f9b" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:05.746210  2907 raft_consensus.cc:2781] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 08:21:05.746613  2144 leader_election.cc:341] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:21:05.746975  2871 leader_election.cc:288] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Requested vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:05.748436  2901 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:05.748622  2901 ts_tablet_manager.cc:1367] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.015s
I20240503 08:21:05.748867  2900 ts_tablet_manager.cc:1398] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.002s
I20240503 08:21:05.749313  2900 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:05.749441  2901 raft_consensus.cc:348] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.749678  2901 raft_consensus.cc:374] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.744755  2902 consensus_queue.cc:260] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.749755  2901 raft_consensus.cc:729] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.750118  2902 ts_tablet_manager.cc:1398] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.001s
I20240503 08:21:05.750020  2901 consensus_queue.cc:260] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.750432  2902 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:05.750607  2901 ts_tablet_manager.cc:1398] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:05.750883  2901 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:05.752557  2901 tablet_bootstrap.cc:654] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.752616  2900 tablet_bootstrap.cc:654] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.753831  2907 raft_consensus.cc:686] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Becoming Leader. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Running, Role: LEADER
I20240503 08:21:05.754267  2907 consensus_queue.cc:237] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.756803  2902 tablet_bootstrap.cc:654] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:05.757455  2901 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:05.757683  2901 ts_tablet_manager.cc:1367] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.004s
I20240503 08:21:05.758107  2900 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:05.758280  2900 ts_tablet_manager.cc:1367] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.002s
I20240503 08:21:05.758491  2901 raft_consensus.cc:348] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.758754  2901 raft_consensus.cc:374] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.758858  2901 raft_consensus.cc:729] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.759126  2901 consensus_queue.cc:260] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.759284  2900 raft_consensus.cc:348] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.759115  2068 catalog_manager.cc:5484] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:05.759544  2900 raft_consensus.cc:374] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.759647  2900 raft_consensus.cc:729] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.759809  2901 ts_tablet_manager.cc:1398] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:05.759963  2900 consensus_queue.cc:260] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.760586  2900 ts_tablet_manager.cc:1398] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:21:05.761719  2902 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:05.761906  2902 ts_tablet_manager.cc:1367] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240503 08:21:05.762725  2902 raft_consensus.cc:348] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.762938  2902 raft_consensus.cc:374] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:05.763046  2902 raft_consensus.cc:729] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:05.763276  2902 consensus_queue.cc:260] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.763794  2902 ts_tablet_manager.cc:1398] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:05.769165  2830 raft_consensus.cc:482] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:05.769315  2830 raft_consensus.cc:504] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.769969  2830 leader_election.cc:288] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:05.770401  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc12dd5961041318d235c7e581238d2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:05.770483  2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc12dd5961041318d235c7e581238d2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:05.770602  2211 raft_consensus.cc:2454] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:05.770692  2479 raft_consensus.cc:2454] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:05.771065  2284 leader_election.cc:302] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:05.771335  2830 raft_consensus.cc:2781] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:05.771454  2830 raft_consensus.cc:482] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:05.771528  2830 raft_consensus.cc:3037] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.773547  2830 raft_consensus.cc:504] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.774087  2830 leader_election.cc:288] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Requested vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:05.774523  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc12dd5961041318d235c7e581238d2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:05.774607  2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc12dd5961041318d235c7e581238d2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:05.774760  2211 raft_consensus.cc:3037] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.774840  2479 raft_consensus.cc:3037] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.777680  2211 raft_consensus.cc:2454] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:05.777680  2479 raft_consensus.cc:2454] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:05.778198  2282 leader_election.cc:302] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:05.778563  2830 raft_consensus.cc:2781] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:05.778934  2830 raft_consensus.cc:686] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:05.779217  2830 consensus_queue.cc:237] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.781989  2064 catalog_manager.cc:5484] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:05.789412  2830 raft_consensus.cc:482] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:05.789625  2830 raft_consensus.cc:504] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.790266  2830 leader_election.cc:288] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:05.790719  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21365f867974e38bef15b340748d8ae" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:05.790830  2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21365f867974e38bef15b340748d8ae" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:05.790992  2211 raft_consensus.cc:2454] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:05.791113  2479 raft_consensus.cc:2454] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:05.791494  2284 leader_election.cc:302] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:05.791790  2830 raft_consensus.cc:2781] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:05.791882  2830 raft_consensus.cc:482] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:05.791986  2830 raft_consensus.cc:3037] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.793826  2830 raft_consensus.cc:504] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.794412  2830 leader_election.cc:288] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Requested vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:05.794821  2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21365f867974e38bef15b340748d8ae" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:05.794852  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21365f867974e38bef15b340748d8ae" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:05.795037  2479 raft_consensus.cc:3037] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.795151  2211 raft_consensus.cc:3037] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.797158  2211 raft_consensus.cc:2454] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:05.797566  2284 leader_election.cc:302] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:05.797588  2479 raft_consensus.cc:2454] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:05.797840  2830 raft_consensus.cc:2781] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:05.798041  2830 raft_consensus.cc:686] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:05.798339  2830 consensus_queue.cc:237] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.801508  2064 catalog_manager.cc:5484] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:05.849951  2906 raft_consensus.cc:482] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:05.850219  2906 raft_consensus.cc:504] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:05.852427  2906 leader_election.cc:288] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:05.861166  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361d2edf4fbb4781b23245a436b49f9b" candidate_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2" is_pre_election: true
I20240503 08:21:05.861524  2349 raft_consensus.cc:2379] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b298d755b2a94cb4bd6e877c0b55a9e4 in current term 1: Already voted for candidate 03bd4573b6f64823bb1b514c106baed6 in this term.
I20240503 08:21:05.861490  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361d2edf4fbb4781b23245a436b49f9b" candidate_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:05.862272  2415 leader_election.cc:302] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2
I20240503 08:21:05.862681  2906 raft_consensus.cc:3037] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:05.865769  2906 raft_consensus.cc:2726] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 08:21:06.185950  2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:06.186345  2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:21:06.192516  2916 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:06.194917  2919 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:21:06.198799  2830 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 08:21:06.254837  2907 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:06.261772  2952 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:06.272276  2830 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:06.278270  2916 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 236.883 ms
  time per row: 0.118442 ms
I20240503 08:21:06.585326  2063 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56190:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 08:21:06.586115  2063 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:21:06.594002  2191 tablet_service.cc:1467] Processing CreateTablet for tablet f2ec11f1dcfa4eb7913d51269a55f7f2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.593997  2327 tablet_service.cc:1467] Processing CreateTablet for tablet f2ec11f1dcfa4eb7913d51269a55f7f2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.594492  2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ec11f1dcfa4eb7913d51269a55f7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.594520  2327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ec11f1dcfa4eb7913d51269a55f7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.594448  2459 tablet_service.cc:1467] Processing CreateTablet for tablet f2ec11f1dcfa4eb7913d51269a55f7f2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.594897  2459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ec11f1dcfa4eb7913d51269a55f7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.595866  2460 tablet_service.cc:1467] Processing CreateTablet for tablet 22d022e74b4b4294a033023b59cc0080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.596267  2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d022e74b4b4294a033023b59cc0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.598575  2458 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc7ea0536ff45dca5da96658c010223 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.598963  2458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc7ea0536ff45dca5da96658c010223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.599555  2190 tablet_service.cc:1467] Processing CreateTablet for tablet 22d022e74b4b4294a033023b59cc0080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.599977  2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d022e74b4b4294a033023b59cc0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.602684  2189 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc7ea0536ff45dca5da96658c010223 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.603222  2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc7ea0536ff45dca5da96658c010223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.607136  2328 tablet_service.cc:1467] Processing CreateTablet for tablet 22d022e74b4b4294a033023b59cc0080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.607335  2329 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc7ea0536ff45dca5da96658c010223 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:06.607609  2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d022e74b4b4294a033023b59cc0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.607789  2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc7ea0536ff45dca5da96658c010223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:06.622743  2984 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:06.623400  2986 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:06.624116  2985 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:06.624841  2984 tablet_bootstrap.cc:654] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.625303  2986 tablet_bootstrap.cc:654] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.625886  2985 tablet_bootstrap.cc:654] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.626812  2984 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:06.626972  2984 ts_tablet_manager.cc:1367] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 08:21:06.627826  2984 raft_consensus.cc:348] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.628072  2984 raft_consensus.cc:374] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.628175  2984 raft_consensus.cc:729] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.628432  2984 consensus_queue.cc:260] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.629032  2984 ts_tablet_manager.cc:1398] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:06.629076  2986 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:06.629212  2986 ts_tablet_manager.cc:1367] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.005s
I20240503 08:21:06.629236  2985 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:06.629338  2984 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:06.629391  2985 ts_tablet_manager.cc:1367] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 08:21:06.629869  2986 raft_consensus.cc:348] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.630036  2986 raft_consensus.cc:374] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.630125  2986 raft_consensus.cc:729] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.630205  2985 raft_consensus.cc:348] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.630427  2985 raft_consensus.cc:374] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.630384  2986 consensus_queue.cc:260] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.630522  2985 raft_consensus.cc:729] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.630924  2986 ts_tablet_manager.cc:1398] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:06.631085  2984 tablet_bootstrap.cc:654] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.631218  2986 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:06.631536  2985 consensus_queue.cc:260] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.632144  2985 ts_tablet_manager.cc:1398] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:21:06.632406  2985 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:06.634140  2986 tablet_bootstrap.cc:654] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.634675  2985 tablet_bootstrap.cc:654] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.637575  2985 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:06.637873  2985 ts_tablet_manager.cc:1367] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240503 08:21:06.639001  2985 raft_consensus.cc:348] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.639411  2985 raft_consensus.cc:374] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.639565  2985 raft_consensus.cc:729] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.639943  2985 consensus_queue.cc:260] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.640692  2985 ts_tablet_manager.cc:1398] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:21:06.641135  2985 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:06.644420  2985 tablet_bootstrap.cc:654] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.646785  2984 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:06.646976  2984 ts_tablet_manager.cc:1367] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.009s
I20240503 08:21:06.647243  2986 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:06.647416  2986 ts_tablet_manager.cc:1367] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.008s
I20240503 08:21:06.647774  2985 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: No bootstrap required, opened a new log
I20240503 08:21:06.647917  2985 ts_tablet_manager.cc:1367] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240503 08:21:06.647879  2984 raft_consensus.cc:348] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.648125  2984 raft_consensus.cc:374] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.648221  2984 raft_consensus.cc:729] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.648192  2986 raft_consensus.cc:348] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.648409  2986 raft_consensus.cc:374] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.648483  2986 raft_consensus.cc:729] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.648510  2984 consensus_queue.cc:260] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.648669  2985 raft_consensus.cc:348] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.648871  2985 raft_consensus.cc:374] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.648957  2985 raft_consensus.cc:729] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.648736  2986 consensus_queue.cc:260] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.649101  2984 ts_tablet_manager.cc:1398] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:06.649191  2985 consensus_queue.cc:260] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.649477  2984 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:06.649834  2985 ts_tablet_manager.cc:1398] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:21:06.651243  2984 tablet_bootstrap.cc:654] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.651608  2986 ts_tablet_manager.cc:1398] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240503 08:21:06.651969  2986 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6: Bootstrap starting.
I20240503 08:21:06.653738  2986 tablet_bootstrap.cc:654] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:06.658145  2984 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: No bootstrap required, opened a new log
I20240503 08:21:06.658257  2986 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6: No bootstrap required, opened a new log
I20240503 08:21:06.658334  2984 ts_tablet_manager.cc:1367] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.007s
I20240503 08:21:06.658417  2986 ts_tablet_manager.cc:1367] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.001s
I20240503 08:21:06.659039  2984 raft_consensus.cc:348] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.659054  2986 raft_consensus.cc:348] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.659231  2984 raft_consensus.cc:374] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.659296  2986 raft_consensus.cc:374] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:06.659327  2984 raft_consensus.cc:729] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.659397  2986 raft_consensus.cc:729] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Initialized, Role: FOLLOWER
I20240503 08:21:06.659566  2984 consensus_queue.cc:260] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.659631  2986 consensus_queue.cc:260] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.660092  2984 ts_tablet_manager.cc:1398] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:06.660688  2986 ts_tablet_manager.cc:1398] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:06.688238  2975 raft_consensus.cc:482] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:06.688436  2975 raft_consensus.cc:504] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.689097  2975 leader_election.cc:288] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:06.689620  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d022e74b4b4294a033023b59cc0080" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:06.689640  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d022e74b4b4294a033023b59cc0080" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:06.689854  2480 raft_consensus.cc:2454] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:06.689875  2211 raft_consensus.cc:2454] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:06.690300  2284 leader_election.cc:302] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:06.690608  2975 raft_consensus.cc:2781] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:06.690704  2975 raft_consensus.cc:482] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:06.690784  2975 raft_consensus.cc:3037] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.692715  2975 raft_consensus.cc:504] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.693241  2975 leader_election.cc:288] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Requested vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:06.693742  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d022e74b4b4294a033023b59cc0080" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:06.693758  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d022e74b4b4294a033023b59cc0080" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:06.693950  2480 raft_consensus.cc:3037] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.693943  2211 raft_consensus.cc:3037] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.696233  2480 raft_consensus.cc:2454] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:06.696265  2211 raft_consensus.cc:2454] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:06.696637  2282 leader_election.cc:302] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:06.696959  2975 raft_consensus.cc:2781] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:06.697109  2975 raft_consensus.cc:686] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:06.697394  2975 consensus_queue.cc:237] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.700026  2063 catalog_manager.cc:5484] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:06.712034  2975 raft_consensus.cc:482] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:06.712257  2975 raft_consensus.cc:504] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.712909  2975 leader_election.cc:288] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:06.713435  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ec11f1dcfa4eb7913d51269a55f7f2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:06.713500  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ec11f1dcfa4eb7913d51269a55f7f2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6" is_pre_election: true
I20240503 08:21:06.713647  2480 raft_consensus.cc:2454] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:06.713747  2211 raft_consensus.cc:2454] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 0.
I20240503 08:21:06.714109  2282 leader_election.cc:302] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:06.714466  2975 raft_consensus.cc:2781] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:06.714553  2975 raft_consensus.cc:482] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:06.714622  2975 raft_consensus.cc:3037] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.716535  2975 raft_consensus.cc:504] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.717097  2975 leader_election.cc:288] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Requested vote from peers 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:06.717664  2211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ec11f1dcfa4eb7913d51269a55f7f2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:06.717650  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ec11f1dcfa4eb7913d51269a55f7f2" candidate_uuid: "2b568c777ade427b9af3dae3395eaaa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:06.717900  2211 raft_consensus.cc:3037] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.717890  2480 raft_consensus.cc:3037] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.720789  2211 raft_consensus.cc:2454] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:06.720789  2480 raft_consensus.cc:2454] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b568c777ade427b9af3dae3395eaaa2 in term 1.
I20240503 08:21:06.721277  2282 leader_election.cc:302] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:06.721590  2975 raft_consensus.cc:2781] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:06.721760  2975 raft_consensus.cc:686] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:06.722046  2975 consensus_queue.cc:237] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.724982  2063 catalog_manager.cc:5484] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:06.853765  2961 raft_consensus.cc:482] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:06.853940  2961 raft_consensus.cc:504] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.854648  2961 leader_election.cc:288] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:06.855129  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc7ea0536ff45dca5da96658c010223" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2" is_pre_election: true
I20240503 08:21:06.855177  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc7ea0536ff45dca5da96658c010223" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" is_pre_election: true
I20240503 08:21:06.855350  2349 raft_consensus.cc:2454] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 0.
I20240503 08:21:06.855446  2480 raft_consensus.cc:2454] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 0.
I20240503 08:21:06.855837  2146 leader_election.cc:302] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2; no voters: 
I20240503 08:21:06.856107  2961 raft_consensus.cc:2781] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:06.856222  2961 raft_consensus.cc:482] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:06.856299  2961 raft_consensus.cc:3037] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.858096  2961 raft_consensus.cc:504] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.858669  2961 leader_election.cc:288] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Requested vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311)
I20240503 08:21:06.859167  2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc7ea0536ff45dca5da96658c010223" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
I20240503 08:21:06.859396  2480 raft_consensus.cc:3037] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.859402  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc7ea0536ff45dca5da96658c010223" candidate_uuid: "03bd4573b6f64823bb1b514c106baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
I20240503 08:21:06.859603  2349 raft_consensus.cc:3037] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:06.861382  2480 raft_consensus.cc:2454] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 1.
I20240503 08:21:06.861424  2349 raft_consensus.cc:2454] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03bd4573b6f64823bb1b514c106baed6 in term 1.
I20240503 08:21:06.861783  2144 leader_election.cc:302] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:06.862090  2961 raft_consensus.cc:2781] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:06.862282  2961 raft_consensus.cc:686] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Becoming Leader. State: Replica: 03bd4573b6f64823bb1b514c106baed6, State: Running, Role: LEADER
I20240503 08:21:06.862600  2961 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } }
I20240503 08:21:06.865324  2063 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129). New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:07.137952  2975 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:07.149883  2975 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:07.169454  2972 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:07.175123  3003 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:21:07.240821  2961 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:07.245334  2961 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:07.271926  2999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:07.272256  2999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:07.280064  2999 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 244.349 ms
  time per row: 0.122174 ms
I20240503 08:21:07.679302  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:34267
--local_ip_for_outbound_sockets=127.2.49.132
--tserver_master_addrs=127.2.49.190:43231
--webserver_port=35399
--webserver_interface=127.2.49.132
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:21:08.046496  3058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:08.046799  3058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:08.046952  3058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:08.055253  3058 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:08.055388  3058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:08.055567  3058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:21:08.066043  3058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:34267
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=35399
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:08.067414  3058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:08.069106  3058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:08.080477  3063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:08.081281  3064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:08.081368  3066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:08.082229  3058 server_base.cc:1034] running on GCE node
I20240503 08:21:08.082818  3058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:08.083828  3058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:08.085103  3058 hybrid_clock.cc:648] HybridClock initialized: now 1714724468085053 us; error 38 us; skew 500 ppm
I20240503 08:21:08.085505  3058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:08.087854  3058 webserver.cc:460] Webserver started at http://127.2.49.132:35399/ using document root <none> and password file <none>
I20240503 08:21:08.088665  3058 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:08.088804  3058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:08.094194  3058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 08:21:08.096959  3072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:08.098506  3058 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 08:21:08.098680  3058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ca50a57f43c9465ba7d4049fd0728361"
format_stamp: "Formatted at 2024-05-03 08:20:58 on dist-test-slave-dz87"
I20240503 08:21:08.099586  3058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:08.162096  3058 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:08.163035  3058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:08.163368  3058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:08.164346  3058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:08.166075  3058 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:08.166183  3058 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:08.166352  3058 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:08.166411  3058 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:08.199858  3058 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:34267
I20240503 08:21:08.199908  3184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:34267 every 8 connection(s)
I20240503 08:21:08.201397  3058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:21:08.204012  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 3058
I20240503 08:21:08.204787  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.133:38143
--local_ip_for_outbound_sockets=127.2.49.133
--tserver_master_addrs=127.2.49.190:43231
--webserver_port=46669
--webserver_interface=127.2.49.133
--builtin_ntp_servers=127.2.49.148:35809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:21:08.216773  3185 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:21:08.217137  3185 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:08.218492  3185 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:21:08.220571  2062 ts_manager.cc:194] Re-registered known tserver with Master: ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132:34267)
I20240503 08:21:08.221554  2062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:38635
W20240503 08:21:08.564652  3189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:08.564990  3189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:08.565137  3189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:08.572912  3189 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:08.573064  3189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:08.573220  3189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.133
I20240503 08:21:08.583078  3189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:35809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.133:38143
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.49.133
--webserver_port=46669
--tserver_master_addrs=127.2.49.190:43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.133
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:08.584472  3189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:08.586313  3189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:08.598647  3195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:08.598701  3194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:08.599243  3197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:08.599594  3189 server_base.cc:1034] running on GCE node
I20240503 08:21:08.600185  3189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:08.601282  3189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:08.602491  3189 hybrid_clock.cc:648] HybridClock initialized: now 1714724468602434 us; error 49 us; skew 500 ppm
I20240503 08:21:08.602859  3189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:08.605278  3189 webserver.cc:460] Webserver started at http://127.2.49.133:46669/ using document root <none> and password file <none>
I20240503 08:21:08.606010  3189 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:08.606125  3189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:08.611634  3189 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.001s
I20240503 08:21:08.614411  3203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:08.615739  3189 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:21:08.615898  3189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "aef8f8b349c44f91ad2c723851a827a0"
format_stamp: "Formatted at 2024-05-03 08:20:59 on dist-test-slave-dz87"
I20240503 08:21:08.616716  3189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:08.645848  3189 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:08.646791  3189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:08.647127  3189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:08.648130  3189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:08.649749  3189 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:08.649839  3189 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:08.649988  3189 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:08.650053  3189 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:08.686187  3189 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.133:38143
I20240503 08:21:08.686201  3315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.133:38143 every 8 connection(s)
I20240503 08:21:08.687944  3189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:21:08.695155  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 3189
I20240503 08:21:08.701397  3316 heartbeater.cc:347] Connected to a master server at 127.2.49.190:43231
I20240503 08:21:08.701813  3316 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:08.702888  3316 heartbeater.cc:511] Master 127.2.49.190:43231 requested a full tablet report, sending...
I20240503 08:21:08.704622  2062 ts_manager.cc:194] Re-registered known tserver with Master: aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143)
I20240503 08:21:08.705505  2062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.133:50769
I20240503 08:21:09.218878  2459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.219502  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:09.221427  2327 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:21:09.226904  3185 heartbeater.cc:503] Master 127.2.49.190:43231 was elected leader, sending a full tablet report...
I20240503 08:21:09.239462  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.254679  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.437471  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:09.439435  2459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.440776  2327 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:21:09.445399  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.471140  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.708587  3316 heartbeater.cc:503] Master 127.2.49.190:43231 was elected leader, sending a full tablet report...
I20240503 08:21:09.742491  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.744593  2459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.745982  2327 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:21:09.746634  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:09.751662  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.943711  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:09.946529  2459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.950471  2327 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:21:09.959277  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:09.959376  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:10.040426  2349 consensus_queue.cc:237] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.043388  2210 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.044456  2477 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.045449  3463 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.047732  3463 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.050079  3436 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.050681  2477 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.051896  2210 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
W20240503 08:21:10.055318  2284 consensus_peers.cc:485] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 -> Peer aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143): Couldn't send request to peer aef8f8b349c44f91ad2c723851a827a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e21365f867974e38bef15b340748d8ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:10.058898  2069 catalog_manager.cc:5484] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:10.072489  2210 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.075431  2349 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.076086  3478 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.076833  2477 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.077443  3478 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.079594  2961 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.081692  2477 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.086126  2070 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:10.088235  2349 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
W20240503 08:21:10.095422  2144 consensus_peers.cc:485] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 -> Peer ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132:34267): Couldn't send request to peer ca50a57f43c9465ba7d4049fd0728361. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4510323bcd68498287476b09df0e8a2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:10.106700  2349 consensus_queue.cc:237] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.109114  2210 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:21:10.109239  2284 consensus_peers.cc:485] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 -> Peer aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143): Couldn't send request to peer aef8f8b349c44f91ad2c723851a827a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22d022e74b4b4294a033023b59cc0080. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:10.109337  2477 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.109652  3003 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.109967  3463 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.112746  3003 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.113343  2210 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.113557  2477 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.118561  2062 catalog_manager.cc:5484] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.130791  2349 consensus_queue.cc:237] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.133394  2477 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.134044  3003 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.134296  2210 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.135838  3003 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.136771  3463 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.137403  2477 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.139273  2210 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.145016  2069 catalog_manager.cc:5484] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
W20240503 08:21:10.145905  2282 consensus_peers.cc:485] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 -> Peer ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132:34267): Couldn't send request to peer ca50a57f43c9465ba7d4049fd0728361. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2ec11f1dcfa4eb7913d51269a55f7f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:10.159878  2210 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.162446  2477 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.163203  2349 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.163133  3483 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.163776  3483 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.166456  3483 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
W20240503 08:21:10.170707  2146 consensus_peers.cc:485] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 -> Peer aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143): Couldn't send request to peer aef8f8b349c44f91ad2c723851a827a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc7ea0536ff45dca5da96658c010223. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:10.172266  2070 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:10.173738  2349 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.182682  2477 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:10.192795  2349 consensus_queue.cc:237] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.195145  2210 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:10.195200  2477 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:21:10.195358  2282 consensus_peers.cc:485] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 -> Peer ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132:34267): Couldn't send request to peer ca50a57f43c9465ba7d4049fd0728361. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc12dd5961041318d235c7e581238d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:10.195757  3500 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.196066  3003 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:10.198846  3003 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.199488  2477 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.203591  2070 catalog_manager.cc:5484] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:10.205173  2210 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:10.461818  3547 ts_tablet_manager.cc:902] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Initiating tablet copy from peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:10.463682  3547 tablet_copy_client.cc:308] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:34357
I20240503 08:21:10.471326  2369 tablet_copy_service.cc:140] P 2b568c777ade427b9af3dae3395eaaa2: Received BeginTabletCopySession request for tablet e21365f867974e38bef15b340748d8ae from peer aef8f8b349c44f91ad2c723851a827a0 ({username='slave'} at 127.2.49.133:59993)
I20240503 08:21:10.471625  2369 tablet_copy_service.cc:161] P 2b568c777ade427b9af3dae3395eaaa2: Beginning new tablet copy session on tablet e21365f867974e38bef15b340748d8ae from peer aef8f8b349c44f91ad2c723851a827a0 at {username='slave'} at 127.2.49.133:59993: session id = aef8f8b349c44f91ad2c723851a827a0-e21365f867974e38bef15b340748d8ae
I20240503 08:21:10.475303  2369 tablet_copy_source_session.cc:205] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:10.481057  3547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21365f867974e38bef15b340748d8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:10.493659  3547 tablet_copy_client.cc:783] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:10.494211  3547 tablet_copy_client.cc:647] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:10.496863  3547 tablet_copy_client.cc:523] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:10.500957  3547 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Bootstrap starting.
I20240503 08:21:10.513250  3547 log.cc:824] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:10.534729  3585 ts_tablet_manager.cc:902] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Initiating tablet copy from peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:10.537308  3585 tablet_copy_client.cc:308] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:34357
I20240503 08:21:10.544787  2369 tablet_copy_service.cc:140] P 2b568c777ade427b9af3dae3395eaaa2: Received BeginTabletCopySession request for tablet f2ec11f1dcfa4eb7913d51269a55f7f2 from peer ca50a57f43c9465ba7d4049fd0728361 ({username='slave'} at 127.2.49.132:52197)
I20240503 08:21:10.545028  2369 tablet_copy_service.cc:161] P 2b568c777ade427b9af3dae3395eaaa2: Beginning new tablet copy session on tablet f2ec11f1dcfa4eb7913d51269a55f7f2 from peer ca50a57f43c9465ba7d4049fd0728361 at {username='slave'} at 127.2.49.132:52197: session id = ca50a57f43c9465ba7d4049fd0728361-f2ec11f1dcfa4eb7913d51269a55f7f2
I20240503 08:21:10.548065  2369 tablet_copy_source_session.cc:205] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:10.552623  3585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ec11f1dcfa4eb7913d51269a55f7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:10.557090  3547 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:10.557574  3547 tablet_bootstrap.cc:492] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Bootstrap complete.
I20240503 08:21:10.557986  3547 ts_tablet_manager.cc:1367] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Time spent bootstrapping tablet: real 0.057s	user 0.048s	sys 0.006s
I20240503 08:21:10.562732  3547 raft_consensus.cc:348] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.563308  3547 raft_consensus.cc:729] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aef8f8b349c44f91ad2c723851a827a0, State: Initialized, Role: LEARNER
I20240503 08:21:10.563932  3547 consensus_queue.cc:260] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.565186  3585 tablet_copy_client.cc:783] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:10.565584  3585 tablet_copy_client.cc:647] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:10.565830  3547 ts_tablet_manager.cc:1398] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0: Time spent starting tablet: real 0.008s	user 0.010s	sys 0.001s
I20240503 08:21:10.566895  2369 tablet_copy_service.cc:342] P 2b568c777ade427b9af3dae3395eaaa2: Request end of tablet copy session aef8f8b349c44f91ad2c723851a827a0-e21365f867974e38bef15b340748d8ae received from {username='slave'} at 127.2.49.133:59993
I20240503 08:21:10.567073  2369 tablet_copy_service.cc:433] P 2b568c777ade427b9af3dae3395eaaa2: ending tablet copy session aef8f8b349c44f91ad2c723851a827a0-e21365f867974e38bef15b340748d8ae on tablet e21365f867974e38bef15b340748d8ae with peer aef8f8b349c44f91ad2c723851a827a0
I20240503 08:21:10.569013  3585 tablet_copy_client.cc:523] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:10.573079  3585 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap starting.
I20240503 08:21:10.582697  3585 log.cc:824] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:10.600975  3547 ts_tablet_manager.cc:902] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: Initiating tablet copy from peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:10.602543  3547 tablet_copy_client.cc:308] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:40211
I20240503 08:21:10.609437  2232 tablet_copy_service.cc:140] P 03bd4573b6f64823bb1b514c106baed6: Received BeginTabletCopySession request for tablet 4dc7ea0536ff45dca5da96658c010223 from peer aef8f8b349c44f91ad2c723851a827a0 ({username='slave'} at 127.2.49.133:49567)
I20240503 08:21:10.609720  2232 tablet_copy_service.cc:161] P 03bd4573b6f64823bb1b514c106baed6: Beginning new tablet copy session on tablet 4dc7ea0536ff45dca5da96658c010223 from peer aef8f8b349c44f91ad2c723851a827a0 at {username='slave'} at 127.2.49.133:49567: session id = aef8f8b349c44f91ad2c723851a827a0-4dc7ea0536ff45dca5da96658c010223
I20240503 08:21:10.613806  2232 tablet_copy_source_session.cc:205] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:10.615509  3597 ts_tablet_manager.cc:902] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: Initiating tablet copy from peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:10.615938  3547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc7ea0536ff45dca5da96658c010223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:10.621438  3585 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:10.621413  3597 tablet_copy_client.cc:308] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:34357
I20240503 08:21:10.622000  3585 tablet_bootstrap.cc:492] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap complete.
I20240503 08:21:10.622390  2368 tablet_copy_service.cc:140] P 2b568c777ade427b9af3dae3395eaaa2: Received BeginTabletCopySession request for tablet 22d022e74b4b4294a033023b59cc0080 from peer aef8f8b349c44f91ad2c723851a827a0 ({username='slave'} at 127.2.49.133:59993)
I20240503 08:21:10.622630  3585 ts_tablet_manager.cc:1367] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Time spent bootstrapping tablet: real 0.050s	user 0.047s	sys 0.005s
I20240503 08:21:10.622627  2368 tablet_copy_service.cc:161] P 2b568c777ade427b9af3dae3395eaaa2: Beginning new tablet copy session on tablet 22d022e74b4b4294a033023b59cc0080 from peer aef8f8b349c44f91ad2c723851a827a0 at {username='slave'} at 127.2.49.133:59993: session id = aef8f8b349c44f91ad2c723851a827a0-22d022e74b4b4294a033023b59cc0080
I20240503 08:21:10.624109  3547 tablet_copy_client.cc:783] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:10.624439  3547 tablet_copy_client.cc:647] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:10.625722  2368 tablet_copy_source_session.cc:205] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:10.626762  3547 tablet_copy_client.cc:523] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:10.627296  3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d022e74b4b4294a033023b59cc0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:10.629211  3585 raft_consensus.cc:348] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.629778  3585 raft_consensus.cc:729] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca50a57f43c9465ba7d4049fd0728361, State: Initialized, Role: LEARNER
I20240503 08:21:10.630432  3585 consensus_queue.cc:260] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.631062  3547 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: Bootstrap starting.
I20240503 08:21:10.632426  3585 ts_tablet_manager.cc:1398] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.004s
I20240503 08:21:10.633380  2368 tablet_copy_service.cc:342] P 2b568c777ade427b9af3dae3395eaaa2: Request end of tablet copy session ca50a57f43c9465ba7d4049fd0728361-f2ec11f1dcfa4eb7913d51269a55f7f2 received from {username='slave'} at 127.2.49.132:52197
I20240503 08:21:10.633530  2368 tablet_copy_service.cc:433] P 2b568c777ade427b9af3dae3395eaaa2: ending tablet copy session ca50a57f43c9465ba7d4049fd0728361-f2ec11f1dcfa4eb7913d51269a55f7f2 on tablet f2ec11f1dcfa4eb7913d51269a55f7f2 with peer ca50a57f43c9465ba7d4049fd0728361
I20240503 08:21:10.633992  3597 tablet_copy_client.cc:783] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:10.634344  3597 tablet_copy_client.cc:647] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:10.636077  3597 tablet_copy_client.cc:523] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:10.639012  3597 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: Bootstrap starting.
I20240503 08:21:10.644394  3585 ts_tablet_manager.cc:902] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: Initiating tablet copy from peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:10.650907  3585 tablet_copy_client.cc:308] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:40211
I20240503 08:21:10.658313  2232 tablet_copy_service.cc:140] P 03bd4573b6f64823bb1b514c106baed6: Received BeginTabletCopySession request for tablet 4510323bcd68498287476b09df0e8a2e from peer ca50a57f43c9465ba7d4049fd0728361 ({username='slave'} at 127.2.49.132:45101)
I20240503 08:21:10.658550  2232 tablet_copy_service.cc:161] P 03bd4573b6f64823bb1b514c106baed6: Beginning new tablet copy session on tablet 4510323bcd68498287476b09df0e8a2e from peer ca50a57f43c9465ba7d4049fd0728361 at {username='slave'} at 127.2.49.132:45101: session id = ca50a57f43c9465ba7d4049fd0728361-4510323bcd68498287476b09df0e8a2e
I20240503 08:21:10.661579  2232 tablet_copy_source_session.cc:205] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:10.663022  3585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4510323bcd68498287476b09df0e8a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:10.669795  3585 tablet_copy_client.cc:783] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:10.670136  3585 tablet_copy_client.cc:647] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:10.672120  3585 tablet_copy_client.cc:523] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:10.675932  3585 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: Bootstrap starting.
I20240503 08:21:10.677693  3597 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:10.678099  3597 tablet_bootstrap.cc:492] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: Bootstrap complete.
I20240503 08:21:10.678390  3597 ts_tablet_manager.cc:1367] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: Time spent bootstrapping tablet: real 0.039s	user 0.038s	sys 0.000s
I20240503 08:21:10.679236  3597 raft_consensus.cc:348] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.679540  3597 raft_consensus.cc:729] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aef8f8b349c44f91ad2c723851a827a0, State: Initialized, Role: LEARNER
I20240503 08:21:10.679764  3597 consensus_queue.cc:260] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.682255  3547 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:10.682397  3597 ts_tablet_manager.cc:1398] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:21:10.682873  3547 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: Bootstrap complete.
I20240503 08:21:10.683156  3547 ts_tablet_manager.cc:1367] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: Time spent bootstrapping tablet: real 0.052s	user 0.039s	sys 0.010s
I20240503 08:21:10.683353  2368 tablet_copy_service.cc:342] P 2b568c777ade427b9af3dae3395eaaa2: Request end of tablet copy session aef8f8b349c44f91ad2c723851a827a0-22d022e74b4b4294a033023b59cc0080 received from {username='slave'} at 127.2.49.133:59993
I20240503 08:21:10.683502  2368 tablet_copy_service.cc:433] P 2b568c777ade427b9af3dae3395eaaa2: ending tablet copy session aef8f8b349c44f91ad2c723851a827a0-22d022e74b4b4294a033023b59cc0080 on tablet 22d022e74b4b4294a033023b59cc0080 with peer aef8f8b349c44f91ad2c723851a827a0
I20240503 08:21:10.684813  3547 raft_consensus.cc:348] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.685101  3547 raft_consensus.cc:729] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aef8f8b349c44f91ad2c723851a827a0, State: Initialized, Role: LEARNER
I20240503 08:21:10.685312  3547 consensus_queue.cc:260] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:10.685988  3547 ts_tablet_manager.cc:1398] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:21:10.686822  2232 tablet_copy_service.cc:342] P 03bd4573b6f64823bb1b514c106baed6: Request end of tablet copy session aef8f8b349c44f91ad2c723851a827a0-4dc7ea0536ff45dca5da96658c010223 received from {username='slave'} at 127.2.49.133:49567
I20240503 08:21:10.686983  2232 tablet_copy_service.cc:433] P 03bd4573b6f64823bb1b514c106baed6: ending tablet copy session aef8f8b349c44f91ad2c723851a827a0-4dc7ea0536ff45dca5da96658c010223 on tablet 4dc7ea0536ff45dca5da96658c010223 with peer aef8f8b349c44f91ad2c723851a827a0
I20240503 08:21:10.695818  3610 ts_tablet_manager.cc:902] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: Initiating tablet copy from peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:10.703171  3610 tablet_copy_client.cc:308] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:34357
I20240503 08:21:10.704206  2368 tablet_copy_service.cc:140] P 2b568c777ade427b9af3dae3395eaaa2: Received BeginTabletCopySession request for tablet cfc12dd5961041318d235c7e581238d2 from peer ca50a57f43c9465ba7d4049fd0728361 ({username='slave'} at 127.2.49.132:52197)
I20240503 08:21:10.704420  2368 tablet_copy_service.cc:161] P 2b568c777ade427b9af3dae3395eaaa2: Beginning new tablet copy session on tablet cfc12dd5961041318d235c7e581238d2 from peer ca50a57f43c9465ba7d4049fd0728361 at {username='slave'} at 127.2.49.132:52197: session id = ca50a57f43c9465ba7d4049fd0728361-cfc12dd5961041318d235c7e581238d2
I20240503 08:21:10.706910  2368 tablet_copy_source_session.cc:205] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:10.708313  3610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc12dd5961041318d235c7e581238d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:10.715296  3610 tablet_copy_client.cc:783] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:10.715675  3610 tablet_copy_client.cc:647] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:10.717590  3610 tablet_copy_client.cc:523] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:10.720350  3610 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap starting.
I20240503 08:21:10.724684  3585 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:10.725085  3585 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: Bootstrap complete.
I20240503 08:21:10.725337  3585 ts_tablet_manager.cc:1367] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: Time spent bootstrapping tablet: real 0.050s	user 0.034s	sys 0.011s
I20240503 08:21:10.726178  3585 raft_consensus.cc:348] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.726414  3585 raft_consensus.cc:729] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca50a57f43c9465ba7d4049fd0728361, State: Initialized, Role: LEARNER
I20240503 08:21:10.726636  3585 consensus_queue.cc:260] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.727360  3585 ts_tablet_manager.cc:1398] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:21:10.728200  2232 tablet_copy_service.cc:342] P 03bd4573b6f64823bb1b514c106baed6: Request end of tablet copy session ca50a57f43c9465ba7d4049fd0728361-4510323bcd68498287476b09df0e8a2e received from {username='slave'} at 127.2.49.132:45101
I20240503 08:21:10.728318  2232 tablet_copy_service.cc:433] P 03bd4573b6f64823bb1b514c106baed6: ending tablet copy session ca50a57f43c9465ba7d4049fd0728361-4510323bcd68498287476b09df0e8a2e on tablet 4510323bcd68498287476b09df0e8a2e with peer ca50a57f43c9465ba7d4049fd0728361
I20240503 08:21:10.758520  3610 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:10.758993  3610 tablet_bootstrap.cc:492] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap complete.
I20240503 08:21:10.759312  3610 ts_tablet_manager.cc:1367] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: Time spent bootstrapping tablet: real 0.039s	user 0.028s	sys 0.011s
I20240503 08:21:10.760056  3610 raft_consensus.cc:348] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.760339  3610 raft_consensus.cc:729] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca50a57f43c9465ba7d4049fd0728361, State: Initialized, Role: LEARNER
I20240503 08:21:10.760571  3610 consensus_queue.cc:260] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:10.761130  3610 ts_tablet_manager.cc:1398] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 08:21:10.763240  2368 tablet_copy_service.cc:342] P 2b568c777ade427b9af3dae3395eaaa2: Request end of tablet copy session ca50a57f43c9465ba7d4049fd0728361-cfc12dd5961041318d235c7e581238d2 received from {username='slave'} at 127.2.49.132:52197
I20240503 08:21:10.763401  2368 tablet_copy_service.cc:433] P 2b568c777ade427b9af3dae3395eaaa2: ending tablet copy session ca50a57f43c9465ba7d4049fd0728361-cfc12dd5961041318d235c7e581238d2 on tablet cfc12dd5961041318d235c7e581238d2 with peer ca50a57f43c9465ba7d4049fd0728361
I20240503 08:21:11.052579  3271 raft_consensus.cc:1203] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:11.110275  3140 raft_consensus.cc:1203] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:11.150219  3140 raft_consensus.cc:1203] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:11.172842  3140 raft_consensus.cc:1203] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:11.187055  3271 raft_consensus.cc:1203] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:11.211258  3271 raft_consensus.cc:1203] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:11.589174  3671 raft_consensus.cc:1050] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2: attempting to promote NON_VOTER aef8f8b349c44f91ad2c723851a827a0 to VOTER
I20240503 08:21:11.590078  3671 consensus_queue.cc:237] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:11.592823  3500 raft_consensus.cc:1050] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2: attempting to promote NON_VOTER ca50a57f43c9465ba7d4049fd0728361 to VOTER
I20240503 08:21:11.593297  3271 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.593549  2477 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.593777  3500 consensus_queue.cc:237] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:11.594380  3742 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.600164  3672 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.602927  2211 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.603412  2480 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.603495  3140 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.604229  3671 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.604643  3500 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.605003  2211 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.604593  3741 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.605530  3743 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.606040  3742 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.610661  3271 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.611174  2477 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.613271  2062 catalog_manager.cc:5484] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.615623  3742 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.619140  2211 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.623359  2210 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.625077  2480 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.625411  3140 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.630422  2070 catalog_manager.cc:5484] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.636319  2349 consensus_queue.cc:237] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:11.636353  2348 consensus_queue.cc:237] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:11.638342  2480 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.638667  3271 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.638909  3741 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.639180  2210 raft_consensus.cc:1261] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.639557  3741 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.639894  3743 consensus_queue.cc:1035] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.640985  3140 raft_consensus.cc:1261] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.641841  3743 consensus_queue.cc:1035] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.642343  3742 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.642709  3671 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.643636  3271 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.646189  2210 raft_consensus.cc:2932] T 22d022e74b4b4294a033023b59cc0080 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.646863  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22d022e74b4b4294a033023b59cc0080 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:11.647215  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2ec11f1dcfa4eb7913d51269a55f7f2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:11.649165  2070 catalog_manager.cc:5484] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.650362  3678 raft_consensus.cc:1050] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: attempting to promote NON_VOTER aef8f8b349c44f91ad2c723851a827a0 to VOTER
I20240503 08:21:11.651463  3678 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:11.651780  3672 raft_consensus.cc:1050] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2: attempting to promote NON_VOTER aef8f8b349c44f91ad2c723851a827a0 to VOTER
I20240503 08:21:11.650467  2480 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.652647  3672 consensus_queue.cc:237] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:11.655735  2477 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.656013  3271 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.656139  2210 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.656512  3672 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.656915  3500 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.657491  3672 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.658265  3140 raft_consensus.cc:2932] T f2ec11f1dcfa4eb7913d51269a55f7f2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.661002  3679 raft_consensus.cc:1050] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: attempting to promote NON_VOTER ca50a57f43c9465ba7d4049fd0728361 to VOTER
I20240503 08:21:11.661801  3679 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:11.662317  2459 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d022e74b4b4294a033023b59cc0080 with delete_type TABLET_DATA_TOMBSTONED (TS b298d755b2a94cb4bd6e877c0b55a9e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47290
I20240503 08:21:11.665874  2070 catalog_manager.cc:5484] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:11.667448  3743 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.668008  2210 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.670895  2191 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ec11f1dcfa4eb7913d51269a55f7f2 with delete_type TABLET_DATA_TOMBSTONED (TS 03bd4573b6f64823bb1b514c106baed6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33328
I20240503 08:21:11.673812  3270 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.677492  3271 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.677596  2067 catalog_manager.cc:5484] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.679023  3482 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.679322  2480 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.679899  3482 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.681365  3761 tablet_replica.cc:330] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: stopping tablet replica
I20240503 08:21:11.682027  3761 raft_consensus.cc:2229] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:11.682658  3761 raft_consensus.cc:2258] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:11.684115  3482 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.684958  3761 ts_tablet_manager.cc:1875] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:11.685037  2348 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.685751  3678 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.689742  2480 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.694864  3743 raft_consensus.cc:1050] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2: attempting to promote NON_VOTER ca50a57f43c9465ba7d4049fd0728361 to VOTER
I20240503 08:21:11.695814  3743 consensus_queue.cc:237] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:11.696246  3271 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.699678  2070 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.702203  3763 tablet_replica.cc:330] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: stopping tablet replica
I20240503 08:21:11.702842  3763 raft_consensus.cc:2229] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:11.704806  3763 raft_consensus.cc:2258] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:11.707711  3761 ts_tablet_manager.cc:1888] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:11.707913  3761 log.cc:1196] T 22d022e74b4b4294a033023b59cc0080 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/22d022e74b4b4294a033023b59cc0080
I20240503 08:21:11.708101  3140 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.708276  2349 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.708916  3482 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.709352  3768 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.709157  2054 catalog_manager.cc:4830] TS b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): tablet 22d022e74b4b4294a033023b59cc0080 (table rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]) successfully deleted
I20240503 08:21:11.711592  2480 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.714226  2347 consensus_queue.cc:237] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:11.716856  3271 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.716929  2477 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.717656  3672 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.717705  3482 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.720571  3500 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.725328  3271 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.731158  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e21365f867974e38bef15b340748d8ae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:11.733740  2478 raft_consensus.cc:1261] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.734073  2062 catalog_manager.cc:5484] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.735661  3500 consensus_queue.cc:1035] T e21365f867974e38bef15b340748d8ae P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.736413  3763 ts_tablet_manager.cc:1875] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:11.740024  2191 tablet_service.cc:1514] Processing DeleteTablet for tablet e21365f867974e38bef15b340748d8ae with delete_type TABLET_DATA_TOMBSTONED (TS 03bd4573b6f64823bb1b514c106baed6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33328
I20240503 08:21:11.741485  2480 raft_consensus.cc:2932] T e21365f867974e38bef15b340748d8ae P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.743435  3482 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.746538  2349 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.749578  2478 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.750435  2067 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.752285  3139 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.752828  3500 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.753341  2210 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.753360  2479 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:11.756481  2348 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:11.756870  2210 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:11.760114  3742 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.760231  3763 ts_tablet_manager.cc:1888] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:11.760398  3763 log.cc:1196] T f2ec11f1dcfa4eb7913d51269a55f7f2 P 03bd4573b6f64823bb1b514c106baed6: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2ec11f1dcfa4eb7913d51269a55f7f2
I20240503 08:21:11.760735  3742 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.761360  3138 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.761391  2480 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.762051  3764 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:11.762861  3764 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.763276  3138 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.765901  3482 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.766423  2478 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.772409  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4510323bcd68498287476b09df0e8a2e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:11.774413  2056 catalog_manager.cc:4830] TS 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211): tablet f2ec11f1dcfa4eb7913d51269a55f7f2 (table rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]) successfully deleted
I20240503 08:21:11.774150  2070 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.774188  3138 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.778425  3763 tablet_replica.cc:330] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: stopping tablet replica
I20240503 08:21:11.779019  2327 tablet_service.cc:1514] Processing DeleteTablet for tablet 4510323bcd68498287476b09df0e8a2e with delete_type TABLET_DATA_TOMBSTONED (TS 2b568c777ade427b9af3dae3395eaaa2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53904
I20240503 08:21:11.779500  3763 raft_consensus.cc:2229] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:11.780139  3763 raft_consensus.cc:2258] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:11.780875  3776 tablet_replica.cc:330] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: stopping tablet replica
I20240503 08:21:11.782276  3776 raft_consensus.cc:2229] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:11.783636  3776 raft_consensus.cc:2258] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:11.784332  3500 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.785449  3776 ts_tablet_manager.cc:1875] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:11.789104  2210 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.792484  3140 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.794888  3776 ts_tablet_manager.cc:1888] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:11.795049  3776 log.cc:1196] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/4510323bcd68498287476b09df0e8a2e
I20240503 08:21:11.794849  2062 catalog_manager.cc:5484] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.795241  3763 ts_tablet_manager.cc:1875] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:11.796182  2056 catalog_manager.cc:4830] TS 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357): tablet 4510323bcd68498287476b09df0e8a2e (table rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]) successfully deleted
I20240503 08:21:11.804841  2348 consensus_queue.cc:237] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:11.805234  2478 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.807484  2210 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.808301  3742 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.809033  3140 raft_consensus.cc:1261] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:11.809542  3500 consensus_queue.cc:1035] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:11.812135  3742 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.813148  2210 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.816234  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cfc12dd5961041318d235c7e581238d2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:11.816530  3140 raft_consensus.cc:2932] T cfc12dd5961041318d235c7e581238d2 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:11.817302  2062 catalog_manager.cc:5484] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:11.821260  3763 ts_tablet_manager.cc:1888] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:11.821473  3763 log.cc:1196] T e21365f867974e38bef15b340748d8ae P 03bd4573b6f64823bb1b514c106baed6: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/e21365f867974e38bef15b340748d8ae
I20240503 08:21:11.822221  2056 catalog_manager.cc:4830] TS 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211): tablet e21365f867974e38bef15b340748d8ae (table rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]) successfully deleted
I20240503 08:21:11.823347  2459 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc12dd5961041318d235c7e581238d2 with delete_type TABLET_DATA_TOMBSTONED (TS b298d755b2a94cb4bd6e877c0b55a9e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47290
I20240503 08:21:11.823737  3761 tablet_replica.cc:330] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: stopping tablet replica
I20240503 08:21:11.823937  3761 raft_consensus.cc:2229] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:11.824199  3761 raft_consensus.cc:2258] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:11.825203  3761 ts_tablet_manager.cc:1875] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:11.830267  3761 ts_tablet_manager.cc:1888] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:11.830410  3761 log.cc:1196] T cfc12dd5961041318d235c7e581238d2 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfc12dd5961041318d235c7e581238d2
I20240503 08:21:11.831058  2054 catalog_manager.cc:4830] TS b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): tablet cfc12dd5961041318d235c7e581238d2 (table rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]) successfully deleted
I20240503 08:21:12.073355  2210 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:12.075261  2348 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:12.075382  3271 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:12.075814  3482 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:12.076121  3768 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:12.078437  3678 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:12.079073  2348 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:12.079123  3271 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:12.083047  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dc7ea0536ff45dca5da96658c010223 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:12.083997  2067 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 5 to 6, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:12.087411  2459 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc7ea0536ff45dca5da96658c010223 with delete_type TABLET_DATA_TOMBSTONED (TS b298d755b2a94cb4bd6e877c0b55a9e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47290
I20240503 08:21:12.087993  3761 tablet_replica.cc:330] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: stopping tablet replica
I20240503 08:21:12.088301  3761 raft_consensus.cc:2229] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:12.088570  3761 raft_consensus.cc:2258] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:12.089506  3761 ts_tablet_manager.cc:1875] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:12.094357  3761 ts_tablet_manager.cc:1888] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:12.094470  3761 log.cc:1196] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dc7ea0536ff45dca5da96658c010223
I20240503 08:21:12.095247  2054 catalog_manager.cc:4830] TS b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): tablet 4dc7ea0536ff45dca5da96658c010223 (table rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]) successfully deleted
I20240503 08:21:12.214394  2327 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:21:12.214740  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:12.225466  2459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:12.228382  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:12.232860  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:12.308523  2348 consensus_queue.cc:237] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:12.311053  2210 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:21:12.311197  2284 consensus_peers.cc:485] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 -> Peer aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143): Couldn't send request to peer aef8f8b349c44f91ad2c723851a827a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6836fefc21b64f9fadcdcd05794fe7bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:12.311313  2480 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:12.311784  3500 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:12.312144  3743 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:12.315129  3742 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.315845  2480 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.319308  2210 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.323168  2070 catalog_manager.cc:5484] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.340903  2348 consensus_queue.cc:237] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:12.343531  2210 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:12.343709  2480 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:21:12.343995  2282 consensus_peers.cc:485] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 -> Peer ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132:34267): Couldn't send request to peer ca50a57f43c9465ba7d4049fd0728361. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50fe0304979f4d668f9b44b954783e17. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:12.344117  3855 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:12.344461  3500 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:12.347604  3857 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:12.348254  2480 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:12.350502  2210 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:12.353914  2067 catalog_manager.cc:5484] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index -1 to 4, NON_VOTER ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } } }
I20240503 08:21:12.366150  2210 consensus_queue.cc:237] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:12.368784  2480 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:12.368885  2348 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:21:12.369134  2146 consensus_peers.cc:485] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 -> Peer aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143): Couldn't send request to peer aef8f8b349c44f91ad2c723851a827a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 361d2edf4fbb4781b23245a436b49f9b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:12.369464  3764 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:12.369772  3482 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:12.373001  3764 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.377087  2480 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.384238  2062 catalog_manager.cc:5484] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.393908  2349 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } }
I20240503 08:21:12.736807  3904 ts_tablet_manager.cc:902] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: Initiating tablet copy from peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:12.737937  3904 tablet_copy_client.cc:308] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:34357
I20240503 08:21:12.738664  2368 tablet_copy_service.cc:140] P 2b568c777ade427b9af3dae3395eaaa2: Received BeginTabletCopySession request for tablet 50fe0304979f4d668f9b44b954783e17 from peer ca50a57f43c9465ba7d4049fd0728361 ({username='slave'} at 127.2.49.132:52197)
I20240503 08:21:12.738852  2368 tablet_copy_service.cc:161] P 2b568c777ade427b9af3dae3395eaaa2: Beginning new tablet copy session on tablet 50fe0304979f4d668f9b44b954783e17 from peer ca50a57f43c9465ba7d4049fd0728361 at {username='slave'} at 127.2.49.132:52197: session id = ca50a57f43c9465ba7d4049fd0728361-50fe0304979f4d668f9b44b954783e17
I20240503 08:21:12.740796  2368 tablet_copy_source_session.cc:205] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:12.742046  3904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe0304979f4d668f9b44b954783e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:12.746712  3904 tablet_copy_client.cc:783] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:12.746959  3904 tablet_copy_client.cc:647] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:12.748757  3904 tablet_copy_client.cc:523] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:12.751243  3904 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap starting.
I20240503 08:21:12.771814  3910 ts_tablet_manager.cc:902] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: Initiating tablet copy from peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357)
I20240503 08:21:12.773805  3910 tablet_copy_client.cc:308] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:34357
I20240503 08:21:12.774626  2368 tablet_copy_service.cc:140] P 2b568c777ade427b9af3dae3395eaaa2: Received BeginTabletCopySession request for tablet 6836fefc21b64f9fadcdcd05794fe7bd from peer aef8f8b349c44f91ad2c723851a827a0 ({username='slave'} at 127.2.49.133:59993)
I20240503 08:21:12.774811  2368 tablet_copy_service.cc:161] P 2b568c777ade427b9af3dae3395eaaa2: Beginning new tablet copy session on tablet 6836fefc21b64f9fadcdcd05794fe7bd from peer aef8f8b349c44f91ad2c723851a827a0 at {username='slave'} at 127.2.49.133:59993: session id = aef8f8b349c44f91ad2c723851a827a0-6836fefc21b64f9fadcdcd05794fe7bd
I20240503 08:21:12.776784  2368 tablet_copy_source_session.cc:205] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:12.778050  3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6836fefc21b64f9fadcdcd05794fe7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:12.781380  3904 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:12.781755  3904 tablet_bootstrap.cc:492] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: Bootstrap complete.
I20240503 08:21:12.782001  3904 ts_tablet_manager.cc:1367] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: Time spent bootstrapping tablet: real 0.031s	user 0.020s	sys 0.007s
I20240503 08:21:12.782711  3904 raft_consensus.cc:348] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:12.782938  3904 raft_consensus.cc:729] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca50a57f43c9465ba7d4049fd0728361, State: Initialized, Role: LEARNER
I20240503 08:21:12.782945  3910 tablet_copy_client.cc:783] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:12.783172  3910 tablet_copy_client.cc:647] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:12.783193  3904 consensus_queue.cc:260] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: true } }
I20240503 08:21:12.784085  3904 ts_tablet_manager.cc:1398] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.004s
I20240503 08:21:12.784782  2368 tablet_copy_service.cc:342] P 2b568c777ade427b9af3dae3395eaaa2: Request end of tablet copy session ca50a57f43c9465ba7d4049fd0728361-50fe0304979f4d668f9b44b954783e17 received from {username='slave'} at 127.2.49.132:52197
I20240503 08:21:12.784919  2368 tablet_copy_service.cc:433] P 2b568c777ade427b9af3dae3395eaaa2: ending tablet copy session ca50a57f43c9465ba7d4049fd0728361-50fe0304979f4d668f9b44b954783e17 on tablet 50fe0304979f4d668f9b44b954783e17 with peer ca50a57f43c9465ba7d4049fd0728361
I20240503 08:21:12.785269  3910 tablet_copy_client.cc:523] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:12.788640  3910 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: Bootstrap starting.
I20240503 08:21:12.819170  3910 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:12.819542  3910 tablet_bootstrap.cc:492] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: Bootstrap complete.
I20240503 08:21:12.819790  3910 ts_tablet_manager.cc:1367] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: Time spent bootstrapping tablet: real 0.031s	user 0.027s	sys 0.004s
I20240503 08:21:12.820544  3910 raft_consensus.cc:348] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:12.820751  3910 raft_consensus.cc:729] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aef8f8b349c44f91ad2c723851a827a0, State: Initialized, Role: LEARNER
I20240503 08:21:12.820995  3910 consensus_queue.cc:260] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:12.821774  3910 ts_tablet_manager.cc:1398] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:21:12.822567  2368 tablet_copy_service.cc:342] P 2b568c777ade427b9af3dae3395eaaa2: Request end of tablet copy session aef8f8b349c44f91ad2c723851a827a0-6836fefc21b64f9fadcdcd05794fe7bd received from {username='slave'} at 127.2.49.133:59993
I20240503 08:21:12.822685  2368 tablet_copy_service.cc:433] P 2b568c777ade427b9af3dae3395eaaa2: ending tablet copy session aef8f8b349c44f91ad2c723851a827a0-6836fefc21b64f9fadcdcd05794fe7bd on tablet 6836fefc21b64f9fadcdcd05794fe7bd with peer aef8f8b349c44f91ad2c723851a827a0
I20240503 08:21:12.875000  3910 ts_tablet_manager.cc:902] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: Initiating tablet copy from peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:12.877657  3910 tablet_copy_client.cc:308] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:40211
I20240503 08:21:12.878508  2232 tablet_copy_service.cc:140] P 03bd4573b6f64823bb1b514c106baed6: Received BeginTabletCopySession request for tablet 361d2edf4fbb4781b23245a436b49f9b from peer aef8f8b349c44f91ad2c723851a827a0 ({username='slave'} at 127.2.49.133:49567)
I20240503 08:21:12.878683  2232 tablet_copy_service.cc:161] P 03bd4573b6f64823bb1b514c106baed6: Beginning new tablet copy session on tablet 361d2edf4fbb4781b23245a436b49f9b from peer aef8f8b349c44f91ad2c723851a827a0 at {username='slave'} at 127.2.49.133:49567: session id = aef8f8b349c44f91ad2c723851a827a0-361d2edf4fbb4781b23245a436b49f9b
I20240503 08:21:12.881548  2232 tablet_copy_source_session.cc:205] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:12.882910  3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361d2edf4fbb4781b23245a436b49f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:12.889088  3910 tablet_copy_client.cc:783] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:12.889341  3910 tablet_copy_client.cc:647] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:12.891165  3910 tablet_copy_client.cc:523] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:12.893787  3910 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: Bootstrap starting.
I20240503 08:21:12.927657  3910 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:12.928076  3910 tablet_bootstrap.cc:492] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: Bootstrap complete.
I20240503 08:21:12.928310  3910 ts_tablet_manager.cc:1367] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: Time spent bootstrapping tablet: real 0.035s	user 0.029s	sys 0.004s
I20240503 08:21:12.928968  3910 raft_consensus.cc:348] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:12.929159  3910 raft_consensus.cc:729] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aef8f8b349c44f91ad2c723851a827a0, State: Initialized, Role: LEARNER
I20240503 08:21:12.929369  3910 consensus_queue.cc:260] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } }
I20240503 08:21:12.929831  3910 ts_tablet_manager.cc:1398] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:21:12.930617  2232 tablet_copy_service.cc:342] P 03bd4573b6f64823bb1b514c106baed6: Request end of tablet copy session aef8f8b349c44f91ad2c723851a827a0-361d2edf4fbb4781b23245a436b49f9b received from {username='slave'} at 127.2.49.133:49567
I20240503 08:21:12.930795  2232 tablet_copy_service.cc:433] P 03bd4573b6f64823bb1b514c106baed6: ending tablet copy session aef8f8b349c44f91ad2c723851a827a0-361d2edf4fbb4781b23245a436b49f9b on tablet 361d2edf4fbb4781b23245a436b49f9b with peer aef8f8b349c44f91ad2c723851a827a0
I20240503 08:21:13.126788  3140 raft_consensus.cc:1203] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:13.247431  3271 raft_consensus.cc:1203] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:13.381947  3271 raft_consensus.cc:1203] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:13.598783  3949 raft_consensus.cc:1050] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2: attempting to promote NON_VOTER ca50a57f43c9465ba7d4049fd0728361 to VOTER
I20240503 08:21:13.599495  3949 consensus_queue.cc:237] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:13.602243  2480 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.602300  2210 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.602887  3857 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.603200  3948 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.603955  3140 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 LEARNER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.604840  3857 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.609421  3857 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.610486  3140 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.613138  2210 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.613694  2480 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.615777  2070 catalog_manager.cc:5484] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 reported cstate change: config changed from index 4 to 5, ca50a57f43c9465ba7d4049fd0728361 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } attrs { replace: true } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.627554  2349 consensus_queue.cc:237] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } }
I20240503 08:21:13.629592  2480 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:13.629668  3140 raft_consensus.cc:1261] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:13.630137  3988 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:13.630487  3949 consensus_queue.cc:1035] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:13.632869  3948 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.633589  3140 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.633548  2480 raft_consensus.cc:2932] T 50fe0304979f4d668f9b44b954783e17 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.635998  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50fe0304979f4d668f9b44b954783e17 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:13.639017  2069 catalog_manager.cc:5484] T 50fe0304979f4d668f9b44b954783e17 P ca50a57f43c9465ba7d4049fd0728361 reported cstate change: config changed from index 5 to 6, VOTER 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } }
I20240503 08:21:13.642254  2191 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fe0304979f4d668f9b44b954783e17 with delete_type TABLET_DATA_TOMBSTONED (TS 03bd4573b6f64823bb1b514c106baed6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33328
I20240503 08:21:13.642936  3997 tablet_replica.cc:330] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: stopping tablet replica
I20240503 08:21:13.643157  3997 raft_consensus.cc:2229] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:13.643568  3997 raft_consensus.cc:2258] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:13.644388  3997 ts_tablet_manager.cc:1875] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:13.648659  3997 ts_tablet_manager.cc:1888] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:13.648797  3997 log.cc:1196] T 50fe0304979f4d668f9b44b954783e17 P 03bd4573b6f64823bb1b514c106baed6: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/50fe0304979f4d668f9b44b954783e17
I20240503 08:21:13.649580  2056 catalog_manager.cc:4830] TS 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211): tablet 50fe0304979f4d668f9b44b954783e17 (table rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]) successfully deleted
I20240503 08:21:13.787058  3949 raft_consensus.cc:1050] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: attempting to promote NON_VOTER aef8f8b349c44f91ad2c723851a827a0 to VOTER
I20240503 08:21:13.787794  3949 consensus_queue.cc:237] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:13.789986  2480 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.790055  3271 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.790055  2210 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Refusing update from remote peer 2b568c777ade427b9af3dae3395eaaa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.790714  3948 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.791054  3988 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.791328  3949 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.795190  3971 raft_consensus.cc:1050] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6: attempting to promote NON_VOTER aef8f8b349c44f91ad2c723851a827a0 to VOTER
I20240503 08:21:13.795040  3949 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.795994  3971 consensus_queue.cc:237] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:13.795743  2210 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.797753  2480 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.803298  2478 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.803085  2062 catalog_manager.cc:5484] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b568c777ade427b9af3dae3395eaaa2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.804090  3970 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.806911  2349 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.807111  3270 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.807991  3271 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 LEARNER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:13.810151  4003 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:13.814061  4003 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 08:21:13.816793  3971 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.823205  3271 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.823820  2478 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.827222  2067 catalog_manager.cc:5484] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 4 to 5, aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:13.832633  2210 consensus_queue.cc:237] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:13.836325  2478 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:13.836938  3271 raft_consensus.cc:1261] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:13.837687  3764 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:13.838182  4003 consensus_queue.cc:1035] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:13.840463  3971 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.841086  2478 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.843590  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 361d2edf4fbb4781b23245a436b49f9b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:13.843319  3271 raft_consensus.cc:2932] T 361d2edf4fbb4781b23245a436b49f9b P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.845949  2062 catalog_manager.cc:5484] T 361d2edf4fbb4781b23245a436b49f9b P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index 5 to 6, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:13.851891  2327 tablet_service.cc:1514] Processing DeleteTablet for tablet 361d2edf4fbb4781b23245a436b49f9b with delete_type TABLET_DATA_TOMBSTONED (TS 2b568c777ade427b9af3dae3395eaaa2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53904
I20240503 08:21:13.853361  4012 tablet_replica.cc:330] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: stopping tablet replica
I20240503 08:21:13.853592  4012 raft_consensus.cc:2229] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:13.853873  4012 pending_rounds.cc:62] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Trying to abort 1 pending ops.
I20240503 08:21:13.854293  4012 pending_rounds.cc:68] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023511444666535936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "361d2edf4fbb4781b23245a436b49f9b" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: NON_VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } } }
I20240503 08:21:13.854703  4012 raft_consensus.cc:2894] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:21:13.854840  4012 raft_consensus.cc:2866] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:21:13.855005  4012 raft_consensus.cc:2258] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:13.855840  4012 ts_tablet_manager.cc:1875] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:13.861003  4012 ts_tablet_manager.cc:1888] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:13.861120  4012 log.cc:1196] T 361d2edf4fbb4781b23245a436b49f9b P 2b568c777ade427b9af3dae3395eaaa2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/361d2edf4fbb4781b23245a436b49f9b
I20240503 08:21:13.861944  2056 catalog_manager.cc:4830] TS 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357): tablet 361d2edf4fbb4781b23245a436b49f9b (table rebalance_test_table_1 [id=a4da5c78d2e54e9caefd16e80cd354b0]) successfully deleted
I20240503 08:21:13.886940  2349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd"
dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33770
I20240503 08:21:13.887272  2349 raft_consensus.cc:595] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:13.906608  2349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd"
dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33786
I20240503 08:21:13.906841  2349 raft_consensus.cc:595] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:14.125559  2349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd"
dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33806
I20240503 08:21:14.125828  2349 raft_consensus.cc:595] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:14.234921  3948 raft_consensus.cc:979] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: : Instructing follower b298d755b2a94cb4bd6e877c0b55a9e4 to start an election
I20240503 08:21:14.235141  3948 raft_consensus.cc:1067] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Signalling peer b298d755b2a94cb4bd6e877c0b55a9e4 to start an election
I20240503 08:21:14.236078  2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd"
dest_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4"
 from {username='slave'} at 127.2.49.130:42173
I20240503 08:21:14.236305  2478 raft_consensus.cc:482] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:21:14.236449  2478 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:14.238459  2478 raft_consensus.cc:504] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:14.239316  2478 leader_election.cc:288] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [CANDIDATE]: Term 2 election: Requested vote from peers 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357), 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211), aef8f8b349c44f91ad2c723851a827a0 (127.2.49.133:38143)
I20240503 08:21:14.239773  2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2b568c777ade427b9af3dae3395eaaa2"
I20240503 08:21:14.239841  2210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "03bd4573b6f64823bb1b514c106baed6"
I20240503 08:21:14.240051  2349 raft_consensus.cc:3032] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:21:14.240146  2349 raft_consensus.cc:729] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Running, Role: LEADER
I20240503 08:21:14.240144  2210 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:14.240496  2349 consensus_queue.cc:260] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:14.241124  2349 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:14.244422  2210 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b298d755b2a94cb4bd6e877c0b55a9e4 in term 2.
I20240503 08:21:14.244947  2349 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b298d755b2a94cb4bd6e877c0b55a9e4 in term 2.
I20240503 08:21:14.245683  2415 leader_election.cc:302] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03bd4573b6f64823bb1b514c106baed6, 2b568c777ade427b9af3dae3395eaaa2, b298d755b2a94cb4bd6e877c0b55a9e4; no voters: 
I20240503 08:21:14.246057  3993 raft_consensus.cc:2781] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:21:14.246691  3993 raft_consensus.cc:686] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 2 LEADER]: Becoming Leader. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Running, Role: LEADER
I20240503 08:21:14.247218  3993 consensus_queue.cc:237] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:14.250945  2067 catalog_manager.cc:5484] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: term changed from 1 to 2, leader changed from 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) to b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:14.252426  3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6836fefc21b64f9fadcdcd05794fe7bd" candidate_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "aef8f8b349c44f91ad2c723851a827a0"
I20240503 08:21:14.252758  3271 raft_consensus.cc:3037] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:14.255362  3271 raft_consensus.cc:2454] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b298d755b2a94cb4bd6e877c0b55a9e4 in term 2.
I20240503 08:21:14.713763  3271 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 2 FOLLOWER]: Refusing update from remote peer b298d755b2a94cb4bd6e877c0b55a9e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:21:14.714460  3993 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:14.736193  2349 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 2 FOLLOWER]: Refusing update from remote peer b298d755b2a94cb4bd6e877c0b55a9e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:21:14.736833  3993 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:14.741711  2210 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 2 FOLLOWER]: Refusing update from remote peer b298d755b2a94cb4bd6e877c0b55a9e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:21:14.742949  3993 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:15.125195  2478 consensus_queue.cc:237] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } }
I20240503 08:21:15.127207  2210 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 2 FOLLOWER]: Refusing update from remote peer b298d755b2a94cb4bd6e877c0b55a9e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:21:15.127207  3271 raft_consensus.cc:1261] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 2 FOLLOWER]: Refusing update from remote peer b298d755b2a94cb4bd6e877c0b55a9e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:21:15.127794  4050 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:15.128134  4065 consensus_queue.cc:1035] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:15.130353  3993 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:15.131021  2210 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P 03bd4573b6f64823bb1b514c106baed6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:15.131093  3271 raft_consensus.cc:2932] T 6836fefc21b64f9fadcdcd05794fe7bd P aef8f8b349c44f91ad2c723851a827a0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } }
I20240503 08:21:15.134320  2054 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6836fefc21b64f9fadcdcd05794fe7bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:15.135885  2067 catalog_manager.cc:5484] T 6836fefc21b64f9fadcdcd05794fe7bd P b298d755b2a94cb4bd6e877c0b55a9e4 reported cstate change: config changed from index 5 to 7, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:15.141209  2327 tablet_service.cc:1514] Processing DeleteTablet for tablet 6836fefc21b64f9fadcdcd05794fe7bd with delete_type TABLET_DATA_TOMBSTONED (TS 2b568c777ade427b9af3dae3395eaaa2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53904
I20240503 08:21:15.141875  4081 tablet_replica.cc:330] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: stopping tablet replica
I20240503 08:21:15.142083  4081 raft_consensus.cc:2229] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:15.142310  4081 raft_consensus.cc:2258] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:15.143282  4081 ts_tablet_manager.cc:1875] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:15.147658  4081 ts_tablet_manager.cc:1888] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:21:15.147778  4081 log.cc:1196] T 6836fefc21b64f9fadcdcd05794fe7bd P 2b568c777ade427b9af3dae3395eaaa2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/6836fefc21b64f9fadcdcd05794fe7bd
I20240503 08:21:15.148514  2056 catalog_manager.cc:4830] TS 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357): tablet 6836fefc21b64f9fadcdcd05794fe7bd (table rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]) successfully deleted
I20240503 08:21:15.278865  2327 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:15.289793  2459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:15.294113  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:15.306291  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:15.318336  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:15.528118  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:15.535450  2327 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:15.555394  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:15.558477  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:15.574560  2459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:15.653898  2210 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } }
I20240503 08:21:15.656734  3140 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:21:15.657256  2146 consensus_peers.cc:485] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 -> Peer 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357): Couldn't send request to peer 2b568c777ade427b9af3dae3395eaaa2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:15.657475  3764 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:15.658779  2478 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:15.659277  4164 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:15.663784  3764 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } } }
I20240503 08:21:15.664453  2478 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } } }
I20240503 08:21:15.670105  2067 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 6 to 7, NON_VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) added. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:15.674221  3138 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } } }
I20240503 08:21:15.707556  2210 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } }
I20240503 08:21:15.709887  3271 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:15.709980  2349 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:21:15.710258  2144 consensus_peers.cc:485] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 -> Peer b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): Couldn't send request to peer b298d755b2a94cb4bd6e877c0b55a9e4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:15.710542  3764 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:15.710880  4164 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:15.713477  4177 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } } }
I20240503 08:21:15.714061  2349 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } } }
I20240503 08:21:15.717257  3271 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } } }
I20240503 08:21:15.719174  2067 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 reported cstate change: config changed from index 6 to 7, NON_VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } } }
I20240503 08:21:16.252118  4213 ts_tablet_manager.cc:902] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Initiating tablet copy from peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:16.253674  4213 tablet_copy_client.cc:308] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:40211
I20240503 08:21:16.254616  2232 tablet_copy_service.cc:140] P 03bd4573b6f64823bb1b514c106baed6: Received BeginTabletCopySession request for tablet 4dc7ea0536ff45dca5da96658c010223 from peer b298d755b2a94cb4bd6e877c0b55a9e4 ({username='slave'} at 127.2.49.131:39365)
I20240503 08:21:16.254763  2232 tablet_copy_service.cc:161] P 03bd4573b6f64823bb1b514c106baed6: Beginning new tablet copy session on tablet 4dc7ea0536ff45dca5da96658c010223 from peer b298d755b2a94cb4bd6e877c0b55a9e4 at {username='slave'} at 127.2.49.131:39365: session id = b298d755b2a94cb4bd6e877c0b55a9e4-4dc7ea0536ff45dca5da96658c010223
I20240503 08:21:16.256727  2232 tablet_copy_source_session.cc:205] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:16.258327  4213 ts_tablet_manager.cc:1875] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:16.263108  4213 ts_tablet_manager.cc:1888] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:16.263442  4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc7ea0536ff45dca5da96658c010223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:16.266130  4218 ts_tablet_manager.cc:902] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Initiating tablet copy from peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211)
I20240503 08:21:16.266139  4213 tablet_copy_client.cc:783] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:16.266448  4213 tablet_copy_client.cc:647] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:16.267910  4218 tablet_copy_client.cc:308] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:40211
I20240503 08:21:16.268654  4213 tablet_copy_client.cc:523] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:16.268635  2232 tablet_copy_service.cc:140] P 03bd4573b6f64823bb1b514c106baed6: Received BeginTabletCopySession request for tablet 4510323bcd68498287476b09df0e8a2e from peer 2b568c777ade427b9af3dae3395eaaa2 ({username='slave'} at 127.2.49.130:43865)
I20240503 08:21:16.268764  2232 tablet_copy_service.cc:161] P 03bd4573b6f64823bb1b514c106baed6: Beginning new tablet copy session on tablet 4510323bcd68498287476b09df0e8a2e from peer 2b568c777ade427b9af3dae3395eaaa2 at {username='slave'} at 127.2.49.130:43865: session id = 2b568c777ade427b9af3dae3395eaaa2-4510323bcd68498287476b09df0e8a2e
I20240503 08:21:16.270776  2232 tablet_copy_source_session.cc:205] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:16.271629  4213 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap starting.
I20240503 08:21:16.271960  4218 ts_tablet_manager.cc:1875] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:16.276400  4218 ts_tablet_manager.cc:1888] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:16.276672  4218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4510323bcd68498287476b09df0e8a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:16.279196  4218 tablet_copy_client.cc:783] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:16.279520  4218 tablet_copy_client.cc:647] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:16.281580  4218 tablet_copy_client.cc:523] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:16.285486  4218 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap starting.
I20240503 08:21:16.321388  4213 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:16.321945  4213 tablet_bootstrap.cc:492] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Bootstrap complete.
I20240503 08:21:16.322279  4213 ts_tablet_manager.cc:1367] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent bootstrapping tablet: real 0.051s	user 0.047s	sys 0.005s
I20240503 08:21:16.323127  4213 raft_consensus.cc:348] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } }
I20240503 08:21:16.323460  4213 raft_consensus.cc:729] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b298d755b2a94cb4bd6e877c0b55a9e4, State: Initialized, Role: LEARNER
I20240503 08:21:16.323693  4213 consensus_queue.cc:260] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: true } }
I20240503 08:21:16.324322  4213 ts_tablet_manager.cc:1398] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:16.325366  2232 tablet_copy_service.cc:342] P 03bd4573b6f64823bb1b514c106baed6: Request end of tablet copy session b298d755b2a94cb4bd6e877c0b55a9e4-4dc7ea0536ff45dca5da96658c010223 received from {username='slave'} at 127.2.49.131:39365
I20240503 08:21:16.325538  2232 tablet_copy_service.cc:433] P 03bd4573b6f64823bb1b514c106baed6: ending tablet copy session b298d755b2a94cb4bd6e877c0b55a9e4-4dc7ea0536ff45dca5da96658c010223 on tablet 4dc7ea0536ff45dca5da96658c010223 with peer b298d755b2a94cb4bd6e877c0b55a9e4
I20240503 08:21:16.335585  4218 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:16.336071  4218 tablet_bootstrap.cc:492] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Bootstrap complete.
I20240503 08:21:16.336604  4218 ts_tablet_manager.cc:1367] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Time spent bootstrapping tablet: real 0.051s	user 0.047s	sys 0.007s
I20240503 08:21:16.337343  4218 raft_consensus.cc:348] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } }
I20240503 08:21:16.337565  4218 raft_consensus.cc:729] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2b568c777ade427b9af3dae3395eaaa2, State: Initialized, Role: LEARNER
I20240503 08:21:16.337755  4218 consensus_queue.cc:260] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: NON_VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: true } }
I20240503 08:21:16.338341  4218 ts_tablet_manager.cc:1398] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:16.339156  2232 tablet_copy_service.cc:342] P 03bd4573b6f64823bb1b514c106baed6: Request end of tablet copy session 2b568c777ade427b9af3dae3395eaaa2-4510323bcd68498287476b09df0e8a2e received from {username='slave'} at 127.2.49.130:43865
I20240503 08:21:16.339315  2232 tablet_copy_service.cc:433] P 03bd4573b6f64823bb1b514c106baed6: ending tablet copy session 2b568c777ade427b9af3dae3395eaaa2-4510323bcd68498287476b09df0e8a2e on tablet 4510323bcd68498287476b09df0e8a2e with peer 2b568c777ade427b9af3dae3395eaaa2
I20240503 08:21:16.769665  2478 raft_consensus.cc:1203] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:21:16.837121  2349 raft_consensus.cc:1203] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 08:21:17.213368  4246 raft_consensus.cc:1050] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6: attempting to promote NON_VOTER 2b568c777ade427b9af3dae3395eaaa2 to VOTER
I20240503 08:21:17.214094  4246 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } }
I20240503 08:21:17.216681  3138 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:17.216689  2478 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:17.216935  2349 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 LEARNER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:17.217310  4268 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:17.217743  4246 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:17.218106  4164 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:17.221693  4268 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.222517  3138 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.222517  2478 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.225278  2349 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.226702  2062 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 7 to 8, 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:17.236440  2210 consensus_queue.cc:237] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } }
I20240503 08:21:17.238303  3138 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:17.238376  2349 raft_consensus.cc:1261] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:17.238862  4246 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:17.239172  4164 consensus_queue.cc:1035] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:17.241397  4268 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.241987  3138 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.242024  2349 raft_consensus.cc:2932] T 4510323bcd68498287476b09df0e8a2e P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.245246  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4510323bcd68498287476b09df0e8a2e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:17.246701  2067 catalog_manager.cc:5484] T 4510323bcd68498287476b09df0e8a2e P ca50a57f43c9465ba7d4049fd0728361 reported cstate change: config changed from index 8 to 9, VOTER b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "ca50a57f43c9465ba7d4049fd0728361" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 34267 } attrs { promote: false } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { promote: false } } }
I20240503 08:21:17.250120  2459 tablet_service.cc:1514] Processing DeleteTablet for tablet 4510323bcd68498287476b09df0e8a2e with delete_type TABLET_DATA_TOMBSTONED (TS b298d755b2a94cb4bd6e877c0b55a9e4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47290
I20240503 08:21:17.250880  4277 tablet_replica.cc:330] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: stopping tablet replica
I20240503 08:21:17.251076  4277 raft_consensus.cc:2229] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:17.251358  4277 raft_consensus.cc:2258] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:17.252137  4277 ts_tablet_manager.cc:1875] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:17.256481  4277 ts_tablet_manager.cc:1888] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:21:17.256608  4277 log.cc:1196] T 4510323bcd68498287476b09df0e8a2e P b298d755b2a94cb4bd6e877c0b55a9e4: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/4510323bcd68498287476b09df0e8a2e
I20240503 08:21:17.257404  2054 catalog_manager.cc:4830] TS b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131:37311): tablet 4510323bcd68498287476b09df0e8a2e (table rebalance_test_table_0 [id=2201ead7bc624ac78d6f80f9a83bbc94]) successfully deleted
I20240503 08:21:17.271992  4268 raft_consensus.cc:1050] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6: attempting to promote NON_VOTER b298d755b2a94cb4bd6e877c0b55a9e4 to VOTER
I20240503 08:21:17.272879  4268 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } }
I20240503 08:21:17.275234  2478 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 LEARNER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:17.275420  3271 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:17.275774  2349 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:17.276019  4164 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:17.276593  4164 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:17.277539  4164 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:17.291137  4245 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.292043  2478 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.296581  3271 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.297168  2070 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 7 to 8, b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:17.296527  2349 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b298d755b2a94cb4bd6e877c0b55a9e4 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "2b568c777ade427b9af3dae3395eaaa2" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 34357 } attrs { replace: true } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.304544  2210 consensus_queue.cc:237] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } }
I20240503 08:21:17.308811  3271 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:17.308852  2478 raft_consensus.cc:1261] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Refusing update from remote peer 03bd4573b6f64823bb1b514c106baed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:17.309506  4246 consensus_queue.cc:1035] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:17.309978  4268 consensus_queue.cc:1037] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 1.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:17.312100  4246 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.313047  2478 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P b298d755b2a94cb4bd6e877c0b55a9e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.315115  3271 raft_consensus.cc:2932] T 4dc7ea0536ff45dca5da96658c010223 P aef8f8b349c44f91ad2c723851a827a0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } } }
I20240503 08:21:17.316241  2056 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dc7ea0536ff45dca5da96658c010223 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:17.318341  2069 catalog_manager.cc:5484] T 4dc7ea0536ff45dca5da96658c010223 P 03bd4573b6f64823bb1b514c106baed6 reported cstate change: config changed from index 8 to 9, VOTER 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "03bd4573b6f64823bb1b514c106baed6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03bd4573b6f64823bb1b514c106baed6" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef8f8b349c44f91ad2c723851a827a0" member_type: VOTER last_known_addr { host: "127.2.49.133" port: 38143 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b298d755b2a94cb4bd6e877c0b55a9e4" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 37311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:17.322382  2327 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc7ea0536ff45dca5da96658c010223 with delete_type TABLET_DATA_TOMBSTONED (TS 2b568c777ade427b9af3dae3395eaaa2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53904
I20240503 08:21:17.323103  4298 tablet_replica.cc:330] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: stopping tablet replica
I20240503 08:21:17.323315  4298 raft_consensus.cc:2229] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:17.323716  4298 raft_consensus.cc:2258] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:17.324690  4298 ts_tablet_manager.cc:1875] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:17.329381  4298 ts_tablet_manager.cc:1888] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:21:17.329493  4298 log.cc:1196] T 4dc7ea0536ff45dca5da96658c010223 P 2b568c777ade427b9af3dae3395eaaa2: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dc7ea0536ff45dca5da96658c010223
I20240503 08:21:17.330241  2056 catalog_manager.cc:4830] TS 2b568c777ade427b9af3dae3395eaaa2 (127.2.49.130:34357): tablet 4dc7ea0536ff45dca5da96658c010223 (table rebalance_test_table_2 [id=65ea90d1368940c3a81898b708c499b8]) successfully deleted
I20240503 08:21:17.613144  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:17.630329  2327 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:17.631803  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:17.643899  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:17.644980  2459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:17.801348  2459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:17.802538  3251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:17.815951  2191 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:17.816152  2327 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:17.829494  3120 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:18.047820  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2129
I20240503 08:21:18.066864  2256 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:18.214535  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2129
W20240503 08:21:18.242281  2414 connection.cc:535] server connection from 127.2.49.129:56643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:21:18.242658  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2266
W20240503 08:21:18.246879  2284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:40211: connect: Connection refused (error 111)
W20240503 08:21:18.248509  2284 consensus_peers.cc:485] T cfc12dd5961041318d235c7e581238d2 P 2b568c777ade427b9af3dae3395eaaa2 -> Peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211): Couldn't send request to peer 03bd4573b6f64823bb1b514c106baed6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:40211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:18.249114  2388 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:21:18.407559  2284 consensus_peers.cc:485] T 22d022e74b4b4294a033023b59cc0080 P 2b568c777ade427b9af3dae3395eaaa2 -> Peer 03bd4573b6f64823bb1b514c106baed6 (127.2.49.129:40211): Couldn't send request to peer 03bd4573b6f64823bb1b514c106baed6. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:40211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:18.408329  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2266
I20240503 08:21:18.435439  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2397
I20240503 08:21:18.441589  2519 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:18.573737  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2397
I20240503 08:21:18.599278  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 3058
I20240503 08:21:18.605439  3179 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:18.723723  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 3058
I20240503 08:21:18.746474  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 3189
I20240503 08:21:18.752662  3310 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:18.868505  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 3189
I20240503 08:21:18.891098  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 2039
I20240503 08:21:18.892277  2100 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:19.037551  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 2039
2024-05-03T08:21:19Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (22838 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240503 08:21:19.098258  2246 test_util.cc:268] Using random seed: 1735843189
I20240503 08:21:19.100301  2246 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:21:19.100373  2246 ts_itest-base.cc:113] --------------
I20240503 08:21:19.100420  2246 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:21:19.100473  2246 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:21:19.100523  2246 ts_itest-base.cc:116] --------------
2024-05-03T08:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:21:19Z Disabled control of system clock
I20240503 08:21:19.155967  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:33685
--webserver_interface=127.2.49.190
--webserver_port=0
--builtin_ntp_servers=127.2.49.148:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.49.190:33685
--location_mapping_cmd=/tmp/dist-test-taskPaEuZ9/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:21:19.521673  4383 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:21:19.522038  4383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:19.522147  4383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:19.522289  4383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:19.529975  4383 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:19.530123  4383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:21:19.530244  4383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:19.530303  4383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:21:19.530397  4383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:21:19.539914  4383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskPaEuZ9/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.2.49.190:33685
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.49.190:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.49.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:19.541354  4383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:19.543152  4383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:19.553936  4389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:19.553968  4388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:19.554543  4391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:19.554795  4383 server_base.cc:1034] running on GCE node
I20240503 08:21:19.555632  4383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:19.557127  4383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:19.558348  4383 hybrid_clock.cc:648] HybridClock initialized: now 1714724479558282 us; error 46 us; skew 500 ppm
I20240503 08:21:19.558684  4383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:19.565109  4383 webserver.cc:460] Webserver started at http://127.2.49.190:41303/ using document root <none> and password file <none>
I20240503 08:21:19.565943  4383 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:19.566087  4383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:19.566423  4383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:21:19.569170  4383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "77cfee7e67b94498931daecca417d44b"
format_stamp: "Formatted at 2024-05-03 08:21:19 on dist-test-slave-dz87"
I20240503 08:21:19.569986  4383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "77cfee7e67b94498931daecca417d44b"
format_stamp: "Formatted at 2024-05-03 08:21:19 on dist-test-slave-dz87"
I20240503 08:21:19.576064  4383 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.000s
I20240503 08:21:19.579874  4397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:19.581287  4383 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 08:21:19.581475  4383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
uuid: "77cfee7e67b94498931daecca417d44b"
format_stamp: "Formatted at 2024-05-03 08:21:19 on dist-test-slave-dz87"
I20240503 08:21:19.581668  4383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:19.603430  4383 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:19.604329  4383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:19.604631  4383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:19.623581  4383 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.190:33685
I20240503 08:21:19.623605  4449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.190:33685 every 8 connection(s)
I20240503 08:21:19.625020  4383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:21:19.628873  4450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:19.633388  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4383
I20240503 08:21:19.633630  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:21:19.639863  4450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b: Bootstrap starting.
I20240503 08:21:19.643548  4450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:19.644946  4450 log.cc:824] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:19.648020  4450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b: No bootstrap required, opened a new log
I20240503 08:21:19.654475  4450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cfee7e67b94498931daecca417d44b" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 33685 } }
I20240503 08:21:19.654884  4450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:19.654984  4450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cfee7e67b94498931daecca417d44b, State: Initialized, Role: FOLLOWER
I20240503 08:21:19.655699  4450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cfee7e67b94498931daecca417d44b" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 33685 } }
I20240503 08:21:19.655958  4450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:21:19.656112  4450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:21:19.656298  4450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:19.658602  4450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cfee7e67b94498931daecca417d44b" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 33685 } }
I20240503 08:21:19.659101  4450 leader_election.cc:302] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cfee7e67b94498931daecca417d44b; no voters: 
I20240503 08:21:19.659665  4450 leader_election.cc:288] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 08:21:19.659819  4455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:19.660543  4455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [term 1 LEADER]: Becoming Leader. State: Replica: 77cfee7e67b94498931daecca417d44b, State: Running, Role: LEADER
I20240503 08:21:19.661127  4455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cfee7e67b94498931daecca417d44b" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 33685 } }
I20240503 08:21:19.661685  4450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:21:19.664372  4457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cfee7e67b94498931daecca417d44b. Latest consensus state: current_term: 1 leader_uuid: "77cfee7e67b94498931daecca417d44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cfee7e67b94498931daecca417d44b" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 33685 } } }
I20240503 08:21:19.664773  4457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [sys.catalog]: This master's current role is: LEADER
I20240503 08:21:19.664681  4456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77cfee7e67b94498931daecca417d44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cfee7e67b94498931daecca417d44b" member_type: VOTER last_known_addr { host: "127.2.49.190" port: 33685 } } }
I20240503 08:21:19.664891  4456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b [sys.catalog]: This master's current role is: LEADER
I20240503 08:21:19.666831  4464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:21:19.671809  4464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:21:19.680493  4464 catalog_manager.cc:1347] Generated new cluster ID: 5b63f175d1c8407d89d53b4791a610b8
I20240503 08:21:19.680630  4464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:21:19.696617  4464 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:21:19.698490  4464 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:21:19.715813  4464 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77cfee7e67b94498931daecca417d44b: Generated new TSK 0
I20240503 08:21:19.717020  4464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:21:19.726861  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.129:0
--local_ip_for_outbound_sockets=127.2.49.129
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--builtin_ntp_servers=127.2.49.148:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:21:20.085880  4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:20.086244  4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:20.086364  4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:20.094000  4474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:20.094132  4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:20.094326  4474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.129
I20240503 08:21:20.104846  4474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.49.129
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.129
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:20.106248  4474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:20.108031  4474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:20.120316  4479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:20.120939  4480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:20.121070  4482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:20.121315  4474 server_base.cc:1034] running on GCE node
I20240503 08:21:20.121918  4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:20.122943  4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:20.124151  4474 hybrid_clock.cc:648] HybridClock initialized: now 1714724480124106 us; error 45 us; skew 500 ppm
I20240503 08:21:20.124488  4474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:20.126870  4474 webserver.cc:460] Webserver started at http://127.2.49.129:45577/ using document root <none> and password file <none>
I20240503 08:21:20.127638  4474 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:20.127756  4474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:20.128070  4474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:21:20.130715  4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
format_stamp: "Formatted at 2024-05-03 08:21:20 on dist-test-slave-dz87"
I20240503 08:21:20.131531  4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
format_stamp: "Formatted at 2024-05-03 08:21:20 on dist-test-slave-dz87"
I20240503 08:21:20.136766  4474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 08:21:20.140277  4488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:20.142061  4474 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:21:20.142315  4474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
format_stamp: "Formatted at 2024-05-03 08:21:20 on dist-test-slave-dz87"
I20240503 08:21:20.142598  4474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:20.158598  4474 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:20.159494  4474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:20.159747  4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:20.160701  4474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:20.162271  4474 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:20.162358  4474 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:20.162442  4474 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:20.162499  4474 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:20.194792  4474 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.129:42693
I20240503 08:21:20.194862  4600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.129:42693 every 8 connection(s)
I20240503 08:21:20.196125  4474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:21:20.200377  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4474
I20240503 08:21:20.200582  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:21:20.204074  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.130:0
--local_ip_for_outbound_sockets=127.2.49.130
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--builtin_ntp_servers=127.2.49.148:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:21:20.209465  4601 heartbeater.cc:347] Connected to a master server at 127.2.49.190:33685
I20240503 08:21:20.209777  4601 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:20.210635  4601 heartbeater.cc:511] Master 127.2.49.190:33685 requested a full tablet report, sending...
I20240503 08:21:20.279615  4414 ts_manager.cc:194] Registered new tserver with Master: 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:20.281385  4414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.129:57509
W20240503 08:21:20.568825  4605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:20.569120  4605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:20.569274  4605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:20.577123  4605 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:20.577255  4605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:20.577435  4605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.130
I20240503 08:21:20.586695  4605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.49.130
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.130
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:20.588088  4605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:20.589769  4605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:20.602605  4611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:20.603108  4612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:20.603231  4614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:20.603755  4605 server_base.cc:1034] running on GCE node
I20240503 08:21:20.604377  4605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:20.605383  4605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:20.606669  4605 hybrid_clock.cc:648] HybridClock initialized: now 1714724480606595 us; error 56 us; skew 500 ppm
I20240503 08:21:20.606992  4605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:20.609388  4605 webserver.cc:460] Webserver started at http://127.2.49.130:39117/ using document root <none> and password file <none>
I20240503 08:21:20.610116  4605 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:20.610252  4605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:20.610569  4605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:21:20.613121  4605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "aaa0775f07344df4993733706f4a5bb3"
format_stamp: "Formatted at 2024-05-03 08:21:20 on dist-test-slave-dz87"
I20240503 08:21:20.613904  4605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "aaa0775f07344df4993733706f4a5bb3"
format_stamp: "Formatted at 2024-05-03 08:21:20 on dist-test-slave-dz87"
I20240503 08:21:20.618960  4605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20240503 08:21:20.622381  4620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:20.624078  4605 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 08:21:20.624260  4605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aaa0775f07344df4993733706f4a5bb3"
format_stamp: "Formatted at 2024-05-03 08:21:20 on dist-test-slave-dz87"
I20240503 08:21:20.624511  4605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:20.647950  4605 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:20.648766  4605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:20.649003  4605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:20.649978  4605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:20.651540  4605 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:20.651628  4605 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:20.651719  4605 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:20.651777  4605 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:20.683670  4605 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.130:44903
I20240503 08:21:20.683725  4732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.130:44903 every 8 connection(s)
I20240503 08:21:20.684995  4605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:21:20.685420  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4605
I20240503 08:21:20.685590  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:21:20.689599  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.131:0
--local_ip_for_outbound_sockets=127.2.49.131
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--builtin_ntp_servers=127.2.49.148:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:21:20.701239  4733 heartbeater.cc:347] Connected to a master server at 127.2.49.190:33685
I20240503 08:21:20.701661  4733 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:20.702857  4733 heartbeater.cc:511] Master 127.2.49.190:33685 requested a full tablet report, sending...
I20240503 08:21:20.765748  4414 ts_manager.cc:194] Registered new tserver with Master: aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:20.766853  4414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.130:39927
W20240503 08:21:21.053102  4736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:21.053424  4736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:21.053563  4736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:21.061189  4736 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:21.061334  4736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:21.061496  4736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.131
I20240503 08:21:21.071435  4736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.49.131
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.131
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:21.072762  4736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:21.074506  4736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:21.086263  4745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:21.086840  4743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:21.086889  4742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:21.087225  4736 server_base.cc:1034] running on GCE node
I20240503 08:21:21.087884  4736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:21.088819  4736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:21.090025  4736 hybrid_clock.cc:648] HybridClock initialized: now 1714724481089967 us; error 45 us; skew 500 ppm
I20240503 08:21:21.090415  4736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:21.092731  4736 webserver.cc:460] Webserver started at http://127.2.49.131:33967/ using document root <none> and password file <none>
I20240503 08:21:21.093502  4736 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:21.093611  4736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:21.093933  4736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:21:21.099942  4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e6f88466bd394def8c59e6d892ba6129"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:21.100821  4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e6f88466bd394def8c59e6d892ba6129"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:21.107254  4736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20240503 08:21:21.111081  4751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:21.112465  4736 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 08:21:21.112612  4736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e6f88466bd394def8c59e6d892ba6129"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:21.112805  4736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:21.144140  4736 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:21.145020  4736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:21.145282  4736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:21.146299  4736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:21.147835  4736 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:21.147922  4736 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:21.148016  4736 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:21.148072  4736 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:21.182969  4736 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.131:44865
I20240503 08:21:21.183038  4863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.131:44865 every 8 connection(s)
I20240503 08:21:21.184428  4736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:21:21.189605  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4736
I20240503 08:21:21.189795  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:21:21.193317  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:0
--local_ip_for_outbound_sockets=127.2.49.132
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--builtin_ntp_servers=127.2.49.148:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:21:21.204872  4864 heartbeater.cc:347] Connected to a master server at 127.2.49.190:33685
I20240503 08:21:21.205354  4864 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:21.206563  4864 heartbeater.cc:511] Master 127.2.49.190:33685 requested a full tablet report, sending...
I20240503 08:21:21.271987  4414 ts_manager.cc:194] Registered new tserver with Master: e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:21.272981  4414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.131:35345
I20240503 08:21:21.284569  4601 heartbeater.cc:503] Master 127.2.49.190:33685 was elected leader, sending a full tablet report...
W20240503 08:21:21.578460  4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:21.578783  4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:21.578923  4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:21.586620  4867 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:21.586777  4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:21.586953  4867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:21:21.596379  4867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=0
--tserver_master_addrs=127.2.49.190:33685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:21.597751  4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:21.599610  4867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:21.612465  4873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:21.612511  4876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:21.612541  4874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:21.613308  4867 server_base.cc:1034] running on GCE node
I20240503 08:21:21.613896  4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:21.614929  4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:21.616138  4867 hybrid_clock.cc:648] HybridClock initialized: now 1714724481616090 us; error 40 us; skew 500 ppm
I20240503 08:21:21.616497  4867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:21.619174  4867 webserver.cc:460] Webserver started at http://127.2.49.132:42111/ using document root <none> and password file <none>
I20240503 08:21:21.619958  4867 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:21.620079  4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:21.620383  4867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:21:21.623350  4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ac08ff0f1c2340389ce4527584c90c29"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:21.624261  4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ac08ff0f1c2340389ce4527584c90c29"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:21.630308  4867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240503 08:21:21.633946  4882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:21.635377  4867 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:21:21.635547  4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ac08ff0f1c2340389ce4527584c90c29"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:21.635764  4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:21.665236  4867 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:21.666195  4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:21.666471  4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:21.667498  4867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:21.669126  4867 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:21.669229  4867 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:21.669349  4867 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:21.669421  4867 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:21.705269  4867 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:45987
I20240503 08:21:21.705369  4995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:45987 every 8 connection(s)
I20240503 08:21:21.706658  4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:21:21.709487  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 4867
I20240503 08:21:21.709650  2246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:21:21.720156  4996 heartbeater.cc:347] Connected to a master server at 127.2.49.190:33685
I20240503 08:21:21.720480  4996 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:21.721323  4996 heartbeater.cc:511] Master 127.2.49.190:33685 requested a full tablet report, sending...
I20240503 08:21:21.769393  4414 ts_manager.cc:194] Registered new tserver with Master: ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987)
I20240503 08:21:21.770087  4733 heartbeater.cc:503] Master 127.2.49.190:33685 was elected leader, sending a full tablet report...
I20240503 08:21:21.770421  4414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:44097
I20240503 08:21:21.782344  2246 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:21:21.799474  2246 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:21:21.799643  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4867
I20240503 08:21:21.808346  4991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:21.892335  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4867
I20240503 08:21:22.275903  4864 heartbeater.cc:503] Master 127.2.49.190:33685 was elected leader, sending a full tablet report...
I20240503 08:21:25.662693  4413 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34130:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 08:21:25.669787  4413 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:21:25.707880  4797 tablet_service.cc:1467] Processing CreateTablet for tablet 903953aea4a84187a6556e4fef71d7f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.709146  4798 tablet_service.cc:1467] Processing CreateTablet for tablet 3175f646e3db4b0fab2f78f2e7e1e973 (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.707764  4799 tablet_service.cc:1467] Processing CreateTablet for tablet a9db45909d7d4de182218001125a8e6a (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.709847  4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903953aea4a84187a6556e4fef71d7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.714375  4534 tablet_service.cc:1467] Processing CreateTablet for tablet 903953aea4a84187a6556e4fef71d7f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.714349  4666 tablet_service.cc:1467] Processing CreateTablet for tablet 903953aea4a84187a6556e4fef71d7f8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.714404  4536 tablet_service.cc:1467] Processing CreateTablet for tablet a9db45909d7d4de182218001125a8e6a (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.715925  4666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903953aea4a84187a6556e4fef71d7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.715929  4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9db45909d7d4de182218001125a8e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.715880  4535 tablet_service.cc:1467] Processing CreateTablet for tablet 3175f646e3db4b0fab2f78f2e7e1e973 (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.716364  4535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175f646e3db4b0fab2f78f2e7e1e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.718227  4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903953aea4a84187a6556e4fef71d7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.718955  4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175f646e3db4b0fab2f78f2e7e1e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.720136  4668 tablet_service.cc:1467] Processing CreateTablet for tablet a9db45909d7d4de182218001125a8e6a (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.720605  4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9db45909d7d4de182218001125a8e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.721436  4667 tablet_service.cc:1467] Processing CreateTablet for tablet 3175f646e3db4b0fab2f78f2e7e1e973 (DEFAULT_TABLE table=rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:25.721933  4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175f646e3db4b0fab2f78f2e7e1e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.722322  4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9db45909d7d4de182218001125a8e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:25.739436  5019 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:25.743037  5020 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:25.743736  5019 tablet_bootstrap.cc:654] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.745123  5019 log.cc:824] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:25.747651  5020 tablet_bootstrap.cc:654] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.749569  5020 log.cc:824] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:25.753293  5023 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:25.755647  5019 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:25.755992  5019 ts_tablet_manager.cc:1367] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.017s	user 0.001s	sys 0.011s
I20240503 08:21:25.756685  5023 tablet_bootstrap.cc:654] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.756753  5020 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:25.757033  5020 ts_tablet_manager.cc:1367] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20240503 08:21:25.757951  5023 log.cc:824] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:25.763866  5023 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:25.764159  5023 ts_tablet_manager.cc:1367] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20240503 08:21:25.765770  5019 raft_consensus.cc:348] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.766305  5019 raft_consensus.cc:374] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.766412  5019 raft_consensus.cc:729] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.766171  5020 raft_consensus.cc:348] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.766660  5020 raft_consensus.cc:374] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.766924  5020 raft_consensus.cc:729] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.767168  5019 consensus_queue.cc:260] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.767614  5020 consensus_queue.cc:260] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.769066  5019 ts_tablet_manager.cc:1398] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.013s	user 0.013s	sys 0.000s
I20240503 08:21:25.769515  5019 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:25.771728  5019 tablet_bootstrap.cc:654] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.771554  5023 raft_consensus.cc:348] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.772017  5023 raft_consensus.cc:374] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.772132  5023 raft_consensus.cc:729] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.772855  5023 consensus_queue.cc:260] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.777196  5023 ts_tablet_manager.cc:1398] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.013s	user 0.013s	sys 0.000s
I20240503 08:21:25.777684  5023 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:25.779582  5023 tablet_bootstrap.cc:654] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.779937  5020 ts_tablet_manager.cc:1398] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.023s	user 0.011s	sys 0.008s
I20240503 08:21:25.781381  5020 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:25.781929  5019 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:25.782123  5019 ts_tablet_manager.cc:1367] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.007s
I20240503 08:21:25.783418  5023 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:25.783294  5019 raft_consensus.cc:348] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.783437  5020 tablet_bootstrap.cc:654] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.783516  5019 raft_consensus.cc:374] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.783563  5023 ts_tablet_manager.cc:1367] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240503 08:21:25.783617  5019 raft_consensus.cc:729] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.783866  5019 consensus_queue.cc:260] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.784431  5019 ts_tablet_manager.cc:1398] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:25.784751  5019 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:25.784768  5023 raft_consensus.cc:348] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.784960  5023 raft_consensus.cc:374] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.785043  5023 raft_consensus.cc:729] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.785257  5023 consensus_queue.cc:260] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.786317  5023 ts_tablet_manager.cc:1398] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:21:25.786660  5023 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:25.786733  5019 tablet_bootstrap.cc:654] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.788983  5023 tablet_bootstrap.cc:654] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.792896  5020 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:25.793074  5020 ts_tablet_manager.cc:1367] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.008s
I20240503 08:21:25.794291  5020 raft_consensus.cc:348] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.794548  5020 raft_consensus.cc:374] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.794667  5020 raft_consensus.cc:729] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.794873  5023 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:25.794950  5020 consensus_queue.cc:260] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.795066  5023 ts_tablet_manager.cc:1367] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.004s
I20240503 08:21:25.795545  5019 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:25.795650  5020 ts_tablet_manager.cc:1398] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:21:25.795722  5019 ts_tablet_manager.cc:1367] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.003s
I20240503 08:21:25.795847  5023 raft_consensus.cc:348] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.796079  5023 raft_consensus.cc:374] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.796141  5020 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:25.796146  5023 raft_consensus.cc:729] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.796429  5023 consensus_queue.cc:260] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.796486  5019 raft_consensus.cc:348] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.796703  5019 raft_consensus.cc:374] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.796787  5019 raft_consensus.cc:729] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.797128  5023 ts_tablet_manager.cc:1398] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:25.797080  5019 consensus_queue.cc:260] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.797612  5019 ts_tablet_manager.cc:1398] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 08:21:25.797878  5020 tablet_bootstrap.cc:654] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:25.806200  5020 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:25.806389  5020 ts_tablet_manager.cc:1367] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.001s
I20240503 08:21:25.806990  5020 raft_consensus.cc:348] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.807193  5020 raft_consensus.cc:374] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:25.807266  5020 raft_consensus.cc:729] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:25.807497  5020 consensus_queue.cc:260] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.808104  5020 ts_tablet_manager.cc:1398] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:25.862299  5028 raft_consensus.cc:482] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:25.862974  5028 raft_consensus.cc:504] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.865253  5028 leader_election.cc:288] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:25.872786  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9db45909d7d4de182218001125a8e6a" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3" is_pre_election: true
I20240503 08:21:25.872836  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9db45909d7d4de182218001125a8e6a" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" is_pre_election: true
I20240503 08:21:25.873399  4556 raft_consensus.cc:2454] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 0.
I20240503 08:21:25.873404  4688 raft_consensus.cc:2454] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 0.
I20240503 08:21:25.874187  4754 leader_election.cc:302] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:25.874645  5028 raft_consensus.cc:2781] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:25.874786  5028 raft_consensus.cc:482] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:25.874881  5028 raft_consensus.cc:3037] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.876988  5028 raft_consensus.cc:504] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.877611  5028 leader_election.cc:288] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 1 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:25.878186  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9db45909d7d4de182218001125a8e6a" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:25.878135  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9db45909d7d4de182218001125a8e6a" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:25.878401  4688 raft_consensus.cc:3037] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.878391  4556 raft_consensus.cc:3037] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.880483  4688 raft_consensus.cc:2454] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 1.
I20240503 08:21:25.880478  4556 raft_consensus.cc:2454] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 1.
I20240503 08:21:25.880955  4755 leader_election.cc:302] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa0775f07344df4993733706f4a5bb3, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:25.881346  5028 raft_consensus.cc:2781] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:25.881742  5028 raft_consensus.cc:686] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Becoming Leader. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Running, Role: LEADER
I20240503 08:21:25.882319  5028 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } }
I20240503 08:21:25.887060  4414 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6f88466bd394def8c59e6d892ba6129 (127.2.49.131). New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } }
W20240503 08:21:25.939853  4865 tablet.cc:2369] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:21:25.940289  4734 tablet.cc:2369] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:21:25.951588  5026 raft_consensus.cc:482] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:25.951817  5026 raft_consensus.cc:504] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
W20240503 08:21:25.952052  4602 tablet.cc:2369] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:21:25.953533  5026 leader_election.cc:288] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:25.960188  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" is_pre_election: true
I20240503 08:21:25.960459  4556 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:25.960548  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:25.960927  4623 leader_election.cc:302] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:25.960980  4819 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:25.961308  5026 raft_consensus.cc:2781] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:25.961432  5026 raft_consensus.cc:482] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:25.961535  5026 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.963676  5026 raft_consensus.cc:504] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.964311  5026 leader_election.cc:288] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:25.964748  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:25.964951  4556 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.964934  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:25.965128  4819 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.966974  4556 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:25.967245  4819 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:25.967386  4623 leader_election.cc:302] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:25.967707  5026 raft_consensus.cc:2781] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:25.968187  5026 raft_consensus.cc:686] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Leader. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:25.968662  5026 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.973794  4414 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa0775f07344df4993733706f4a5bb3 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:25.994127  5026 raft_consensus.cc:482] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:25.994411  5026 raft_consensus.cc:504] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.995028  5026 leader_election.cc:288] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:25.995615  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" is_pre_election: true
I20240503 08:21:25.995625  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:25.995838  4556 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:25.995851  4819 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:25.996258  4623 leader_election.cc:302] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:25.996623  5026 raft_consensus.cc:2781] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:25.996740  5026 raft_consensus.cc:482] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:25.996829  5026 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:25.998904  5026 raft_consensus.cc:504] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:25.999600  5026 leader_election.cc:288] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:26.000299  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:26.000583  4819 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:26.000665  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:26.000885  4556 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:26.003695  4819 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:26.003888  4556 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:26.004343  4621 leader_election.cc:302] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa0775f07344df4993733706f4a5bb3, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:26.004681  5026 raft_consensus.cc:2781] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:26.004863  5026 raft_consensus.cc:686] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Leader. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:26.005163  5026 consensus_queue.cc:237] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.008020  4412 catalog_manager.cc:5484] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa0775f07344df4993733706f4a5bb3 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:26.378769  5026 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:26.389261  5026 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:26.455964  5043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:26.456329  5043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:21:26.459589  5049 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:26.466899  5049 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:26.467319  5043 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:21:26.486922  5047 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:26.492835  5040 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 276.275 ms
  time per row: 0.138138 ms
I20240503 08:21:26.901538  4409 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34130:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 08:21:26.902444  4409 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:21:26.911430  4667 tablet_service.cc:1467] Processing CreateTablet for tablet ef9404adc1c94a719047974f0f1c2a22 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.911737  4535 tablet_service.cc:1467] Processing CreateTablet for tablet ef9404adc1c94a719047974f0f1c2a22 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.911957  4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9404adc1c94a719047974f0f1c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.912181  4535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9404adc1c94a719047974f0f1c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.912103  4534 tablet_service.cc:1467] Processing CreateTablet for tablet 6950bc46aa234575ad15e3b34a7e4e3d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.912449  4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6950bc46aa234575ad15e3b34a7e4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.913460  4668 tablet_service.cc:1467] Processing CreateTablet for tablet e823e4ca6d5e49a68456ed1da6a82b59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.913858  4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e823e4ca6d5e49a68456ed1da6a82b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.914650  4666 tablet_service.cc:1467] Processing CreateTablet for tablet 6950bc46aa234575ad15e3b34a7e4e3d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.915052  4666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6950bc46aa234575ad15e3b34a7e4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.919121  4797 tablet_service.cc:1467] Processing CreateTablet for tablet ef9404adc1c94a719047974f0f1c2a22 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.919593  4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9404adc1c94a719047974f0f1c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.922788  4799 tablet_service.cc:1467] Processing CreateTablet for tablet e823e4ca6d5e49a68456ed1da6a82b59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.923403  4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e823e4ca6d5e49a68456ed1da6a82b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.926523  4798 tablet_service.cc:1467] Processing CreateTablet for tablet 6950bc46aa234575ad15e3b34a7e4e3d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.927073  4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6950bc46aa234575ad15e3b34a7e4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.931237  4536 tablet_service.cc:1467] Processing CreateTablet for tablet e823e4ca6d5e49a68456ed1da6a82b59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:26.931748  5107 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:26.931911  4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e823e4ca6d5e49a68456ed1da6a82b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:26.932791  5108 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:26.934796  5108 tablet_bootstrap.cc:654] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.943873  5108 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:26.944160  5108 ts_tablet_manager.cc:1367] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240503 08:21:26.945161  5108 raft_consensus.cc:348] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.945796  5108 raft_consensus.cc:374] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.945963  5108 raft_consensus.cc:729] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.946077  5107 tablet_bootstrap.cc:654] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.946430  5108 consensus_queue.cc:260] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.947626  5106 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:26.948920  5108 ts_tablet_manager.cc:1398] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 08:21:26.949291  5108 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:26.949465  5106 tablet_bootstrap.cc:654] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.952450  5108 tablet_bootstrap.cc:654] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.955602  5107 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:26.955793  5107 ts_tablet_manager.cc:1367] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.000s
I20240503 08:21:26.956044  5108 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:26.956115  5106 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:26.956218  5108 ts_tablet_manager.cc:1367] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.005s
I20240503 08:21:26.956275  5106 ts_tablet_manager.cc:1367] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.006s
I20240503 08:21:26.956697  5107 raft_consensus.cc:348] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.956933  5107 raft_consensus.cc:374] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.957057  5107 raft_consensus.cc:729] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.956956  5106 raft_consensus.cc:348] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.956992  5108 raft_consensus.cc:348] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.957157  5106 raft_consensus.cc:374] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.957230  5108 raft_consensus.cc:374] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.957249  5106 raft_consensus.cc:729] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.957315  5108 raft_consensus.cc:729] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.957319  5107 consensus_queue.cc:260] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.957556  5108 consensus_queue.cc:260] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.957509  5106 consensus_queue.cc:260] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.958109  5106 ts_tablet_manager.cc:1398] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:21:26.958211  5108 ts_tablet_manager.cc:1398] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:21:26.958415  5106 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:26.958516  5108 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:26.958557  5107 ts_tablet_manager.cc:1398] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 08:21:26.958812  5107 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:26.960147  5108 tablet_bootstrap.cc:654] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.960377  5107 tablet_bootstrap.cc:654] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.964406  5106 tablet_bootstrap.cc:654] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.972487  5106 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:26.972694  5106 ts_tablet_manager.cc:1367] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.000s
I20240503 08:21:26.973634  5108 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:26.973526  5106 raft_consensus.cc:348] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.973691  5107 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:26.973768  5106 raft_consensus.cc:374] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.973798  5108 ts_tablet_manager.cc:1367] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.015s	user 0.010s	sys 0.003s
I20240503 08:21:26.973865  5107 ts_tablet_manager.cc:1367] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.009s
I20240503 08:21:26.973863  5106 raft_consensus.cc:729] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.974152  5106 consensus_queue.cc:260] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.974610  5108 raft_consensus.cc:348] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.974823  5108 raft_consensus.cc:374] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.974727  5107 raft_consensus.cc:348] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.974890  5106 ts_tablet_manager.cc:1398] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:26.974910  5108 raft_consensus.cc:729] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.974962  5107 raft_consensus.cc:374] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.975041  5107 raft_consensus.cc:729] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.975179  5108 consensus_queue.cc:260] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.975287  5107 consensus_queue.cc:260] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.975746  5108 ts_tablet_manager.cc:1398] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:26.976449  5107 ts_tablet_manager.cc:1398] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:26.976775  5107 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:26.977121  5106 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:26.978577  5107 tablet_bootstrap.cc:654] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.981045  5106 tablet_bootstrap.cc:654] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:26.981529  5107 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:26.981706  5107 ts_tablet_manager.cc:1367] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 08:21:26.982561  5107 raft_consensus.cc:348] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.982782  5107 raft_consensus.cc:374] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.982887  5107 raft_consensus.cc:729] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.983114  5106 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:26.983268  5106 ts_tablet_manager.cc:1367] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.001s
I20240503 08:21:26.983165  5107 consensus_queue.cc:260] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.983727  5107 ts_tablet_manager.cc:1398] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:21:26.984067  5106 raft_consensus.cc:348] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.984220  5106 raft_consensus.cc:374] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:26.984302  5106 raft_consensus.cc:729] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:26.984535  5106 consensus_queue.cc:260] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:26.984993  5106 ts_tablet_manager.cc:1398] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:27.124106  5040 raft_consensus.cc:482] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:27.124450  5040 raft_consensus.cc:504] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.125360  5040 leader_election.cc:288] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:27.126006  4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" is_pre_election: true
I20240503 08:21:27.126287  4555 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:27.126369  4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:27.126610  4818 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:27.127091  4621 leader_election.cc:302] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa0775f07344df4993733706f4a5bb3, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:27.127382  5040 raft_consensus.cc:2781] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:27.127487  5040 raft_consensus.cc:482] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:27.127602  5040 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.129504  5040 raft_consensus.cc:504] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.130203  5040 leader_election.cc:288] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:27.130712  4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:27.130749  4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:27.130920  4818 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.131011  4555 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.133342  4818 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:27.133425  4555 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:27.133790  4621 leader_election.cc:302] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa0775f07344df4993733706f4a5bb3, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:27.134095  5040 raft_consensus.cc:2781] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:27.134346  5040 raft_consensus.cc:686] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Leader. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:27.134634  5040 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.137434  4406 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa0775f07344df4993733706f4a5bb3 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:27.161998  5044 raft_consensus.cc:482] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:27.162281  5044 raft_consensus.cc:504] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.164083  5044 leader_election.cc:288] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:27.182713  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9404adc1c94a719047974f0f1c2a22" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3" is_pre_election: true
I20240503 08:21:27.183038  4688 raft_consensus.cc:2454] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:27.183002  4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9404adc1c94a719047974f0f1c2a22" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:27.183275  4818 raft_consensus.cc:2454] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:27.183614  4492 leader_election.cc:302] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:27.184088  5044 raft_consensus.cc:2781] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:27.184219  5044 raft_consensus.cc:482] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:27.184311  5044 raft_consensus.cc:3037] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.186717  5044 raft_consensus.cc:504] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.187451  5044 leader_election.cc:288] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:27.187963  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9404adc1c94a719047974f0f1c2a22" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:27.187978  4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9404adc1c94a719047974f0f1c2a22" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:27.188179  4688 raft_consensus.cc:3037] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.188190  4818 raft_consensus.cc:3037] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.190459  4688 raft_consensus.cc:2454] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:27.190915  4492 leader_election.cc:302] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:27.191150  4818 raft_consensus.cc:2454] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:27.191205  5044 raft_consensus.cc:2781] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:27.194515  5044 raft_consensus.cc:686] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Becoming Leader. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Running, Role: LEADER
I20240503 08:21:27.195468  5044 consensus_queue.cc:237] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.199723  4406 catalog_manager.cc:5484] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129). New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:27.344758  5044 raft_consensus.cc:482] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:27.344983  5044 raft_consensus.cc:504] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.345613  5044 leader_election.cc:288] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:27.346199  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e823e4ca6d5e49a68456ed1da6a82b59" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3" is_pre_election: true
I20240503 08:21:27.346199  4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e823e4ca6d5e49a68456ed1da6a82b59" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:27.346436  4688 raft_consensus.cc:2454] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:27.346458  4818 raft_consensus.cc:2454] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:27.346957  4489 leader_election.cc:302] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:27.347255  5044 raft_consensus.cc:2781] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:27.347368  5044 raft_consensus.cc:482] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:27.347451  5044 raft_consensus.cc:3037] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.349260  5044 raft_consensus.cc:504] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.349838  5044 leader_election.cc:288] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:27.350365  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e823e4ca6d5e49a68456ed1da6a82b59" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:27.350376  4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e823e4ca6d5e49a68456ed1da6a82b59" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:27.350546  4688 raft_consensus.cc:3037] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.350555  4818 raft_consensus.cc:3037] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:27.352392  4818 raft_consensus.cc:2454] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:27.352521  4688 raft_consensus.cc:2454] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:27.352830  4489 leader_election.cc:302] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:27.353109  5044 raft_consensus.cc:2781] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:27.353284  5044 raft_consensus.cc:686] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Becoming Leader. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Running, Role: LEADER
I20240503 08:21:27.353529  5044 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:27.356192  4406 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129). New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:27.570225  5040 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:27.577363  5047 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:27.749810  5044 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:27.752916  5044 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:27.758634  5044 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:27.771529  5123 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:27.785274  5126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:27.785683  5126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:27.793982  5126 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 215.638 ms
  time per row: 0.107819 ms
I20240503 08:21:28.169678  4410 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34130:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 08:21:28.170526  4410 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:21:28.179092  4536 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1be98d63454a3396be3d44cc4e9578 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.179632  4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1be98d63454a3396be3d44cc4e9578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.179687  4799 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1be98d63454a3396be3d44cc4e9578 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.180017  4534 tablet_service.cc:1467] Processing CreateTablet for tablet 6a85c62c6b934706991a68b034430c3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.180186  4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1be98d63454a3396be3d44cc4e9578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.180418  4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a85c62c6b934706991a68b034430c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.180630  4667 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1be98d63454a3396be3d44cc4e9578 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.181067  4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1be98d63454a3396be3d44cc4e9578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.186389  4666 tablet_service.cc:1467] Processing CreateTablet for tablet 6a85c62c6b934706991a68b034430c3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.186820  4666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a85c62c6b934706991a68b034430c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.190601  4668 tablet_service.cc:1467] Processing CreateTablet for tablet 0044f42850c84c95a6543052a258ee36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.191076  4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0044f42850c84c95a6543052a258ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.191356  5190 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:28.195170  4798 tablet_service.cc:1467] Processing CreateTablet for tablet 6a85c62c6b934706991a68b034430c3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.195611  4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a85c62c6b934706991a68b034430c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.198593  4797 tablet_service.cc:1467] Processing CreateTablet for tablet 0044f42850c84c95a6543052a258ee36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.199004  4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0044f42850c84c95a6543052a258ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.204617  4536 tablet_service.cc:1467] Processing CreateTablet for tablet 0044f42850c84c95a6543052a258ee36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:21:28.205186  4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0044f42850c84c95a6543052a258ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:28.210382  5190 tablet_bootstrap.cc:654] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.214730  5192 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:28.216747  5192 tablet_bootstrap.cc:654] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.224558  5191 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:28.225144  5190 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:28.225337  5190 ts_tablet_manager.cc:1367] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.034s	user 0.004s	sys 0.000s
I20240503 08:21:28.226137  5190 raft_consensus.cc:348] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.226380  5190 raft_consensus.cc:374] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.226465  5190 raft_consensus.cc:729] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.226703  5190 consensus_queue.cc:260] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.227038  5191 tablet_bootstrap.cc:654] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.227160  5192 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:28.227315  5192 ts_tablet_manager.cc:1367] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.000s
I20240503 08:21:28.227466  5190 ts_tablet_manager.cc:1398] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 08:21:28.227749  5190 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:28.228164  5192 raft_consensus.cc:348] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.228364  5192 raft_consensus.cc:374] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.228452  5192 raft_consensus.cc:729] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.228725  5192 consensus_queue.cc:260] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.229342  5190 tablet_bootstrap.cc:654] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.231815  5192 ts_tablet_manager.cc:1398] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 08:21:28.232887  5190 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:28.233055  5190 ts_tablet_manager.cc:1367] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240503 08:21:28.233820  5190 raft_consensus.cc:348] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.234040  5190 raft_consensus.cc:374] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.234130  5190 raft_consensus.cc:729] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.234414  5190 consensus_queue.cc:260] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.235029  5190 ts_tablet_manager.cc:1398] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:28.235388  5190 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:28.237066  5190 tablet_bootstrap.cc:654] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.240784  5192 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:28.242770  5192 tablet_bootstrap.cc:654] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.245932  5190 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: No bootstrap required, opened a new log
I20240503 08:21:28.246086  5190 ts_tablet_manager.cc:1367] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.007s
I20240503 08:21:28.246908  5190 raft_consensus.cc:348] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.247115  5190 raft_consensus.cc:374] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.247202  5190 raft_consensus.cc:729] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.247453  5190 consensus_queue.cc:260] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.248109  5190 ts_tablet_manager.cc:1398] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 08:21:28.249289  5191 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:28.249492  5191 ts_tablet_manager.cc:1367] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.013s
I20240503 08:21:28.250357  5191 raft_consensus.cc:348] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.250658  5191 raft_consensus.cc:374] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.250784  5191 raft_consensus.cc:729] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.251049  5191 consensus_queue.cc:260] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.251721  5191 ts_tablet_manager.cc:1398] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 08:21:28.252010  5191 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:28.253571  5191 tablet_bootstrap.cc:654] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.255064  5192 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:28.255304  5192 ts_tablet_manager.cc:1367] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.006s
I20240503 08:21:28.256202  5192 raft_consensus.cc:348] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.256681  5192 raft_consensus.cc:374] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.256815  5192 raft_consensus.cc:729] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.257201  5192 consensus_queue.cc:260] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.257966  5192 ts_tablet_manager.cc:1398] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 08:21:28.258296  5192 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:28.258566  5191 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:28.258721  5191 ts_tablet_manager.cc:1367] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240503 08:21:28.259548  5191 raft_consensus.cc:348] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.259778  5191 raft_consensus.cc:374] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.259865  5191 raft_consensus.cc:729] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.260038  5192 tablet_bootstrap.cc:654] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.260154  5191 consensus_queue.cc:260] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.260720  5191 ts_tablet_manager.cc:1398] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:28.261332  5191 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: Bootstrap starting.
I20240503 08:21:28.264042  5191 tablet_bootstrap.cc:654] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: Neither blocks nor log segments found. Creating new log.
I20240503 08:21:28.266404  5191 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: No bootstrap required, opened a new log
I20240503 08:21:28.266398  5192 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: No bootstrap required, opened a new log
I20240503 08:21:28.266554  5191 ts_tablet_manager.cc:1367] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240503 08:21:28.266566  5192 ts_tablet_manager.cc:1367] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.003s
I20240503 08:21:28.267382  5191 raft_consensus.cc:348] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.267381  5192 raft_consensus.cc:348] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.267606  5191 raft_consensus.cc:374] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.267609  5192 raft_consensus.cc:374] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:21:28.267707  5191 raft_consensus.cc:729] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.267707  5192 raft_consensus.cc:729] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: FOLLOWER
I20240503 08:21:28.267969  5192 consensus_queue.cc:260] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.267952  5191 consensus_queue.cc:260] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.268519  5191 ts_tablet_manager.cc:1398] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:28.268545  5192 ts_tablet_manager.cc:1398] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 08:21:28.282122  5040 raft_consensus.cc:482] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:28.282382  5040 raft_consensus.cc:504] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.283057  5040 leader_election.cc:288] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:28.283624  4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" is_pre_election: true
I20240503 08:21:28.283661  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:28.283869  4819 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:28.283890  4555 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 0.
I20240503 08:21:28.284348  4621 leader_election.cc:302] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:28.284660  5040 raft_consensus.cc:2781] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:28.284747  5040 raft_consensus.cc:482] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:28.284857  5040 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.287050  5040 raft_consensus.cc:504] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.287675  5040 leader_election.cc:288] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:28.288033  4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:28.288228  4555 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.288224  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "aaa0775f07344df4993733706f4a5bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:28.288456  4819 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.290125  4555 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:28.290458  4819 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa0775f07344df4993733706f4a5bb3 in term 1.
I20240503 08:21:28.290575  4623 leader_election.cc:302] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:28.290892  5040 raft_consensus.cc:2781] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:28.291074  5040 raft_consensus.cc:686] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Leader. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:28.291326  5040 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.294252  4414 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa0775f07344df4993733706f4a5bb3 (127.2.49.130). New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:28.300839  5044 raft_consensus.cc:482] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:28.301010  5044 raft_consensus.cc:504] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.301604  5044 leader_election.cc:288] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:28.302027  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0044f42850c84c95a6543052a258ee36" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3" is_pre_election: true
I20240503 08:21:28.302193  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0044f42850c84c95a6543052a258ee36" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:28.302312  4688 raft_consensus.cc:2454] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:28.302433  4819 raft_consensus.cc:2454] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:28.302791  4492 leader_election.cc:302] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:28.303095  5044 raft_consensus.cc:2781] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:28.303189  5044 raft_consensus.cc:482] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:28.303294  5044 raft_consensus.cc:3037] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.305227  5044 raft_consensus.cc:504] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.305866  5044 leader_election.cc:288] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:28.306246  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0044f42850c84c95a6543052a258ee36" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:28.306481  4688 raft_consensus.cc:3037] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.306424  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0044f42850c84c95a6543052a258ee36" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:28.306660  4819 raft_consensus.cc:3037] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.308322  4688 raft_consensus.cc:2454] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:28.308511  4819 raft_consensus.cc:2454] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:28.308701  4492 leader_election.cc:302] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:28.308991  5044 raft_consensus.cc:2781] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:28.309141  5044 raft_consensus.cc:686] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Becoming Leader. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Running, Role: LEADER
I20240503 08:21:28.309386  5044 consensus_queue.cc:237] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.312085  4414 catalog_manager.cc:5484] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129). New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:28.444885  5044 raft_consensus.cc:482] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:21:28.445052  5044 raft_consensus.cc:504] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.445605  5044 leader_election.cc:288] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:28.446090  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a85c62c6b934706991a68b034430c3b" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129" is_pre_election: true
I20240503 08:21:28.446202  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a85c62c6b934706991a68b034430c3b" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3" is_pre_election: true
I20240503 08:21:28.446419  4819 raft_consensus.cc:2454] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:28.446434  4688 raft_consensus.cc:2454] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 0.
I20240503 08:21:28.446841  4489 leader_election.cc:302] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:28.447130  5044 raft_consensus.cc:2781] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:21:28.447235  5044 raft_consensus.cc:482] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:21:28.447340  5044 raft_consensus.cc:3037] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.449380  5044 raft_consensus.cc:504] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.449966  5044 leader_election.cc:288] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:28.450397  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a85c62c6b934706991a68b034430c3b" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:28.450444  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a85c62c6b934706991a68b034430c3b" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:28.450577  4688 raft_consensus.cc:3037] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.450627  4819 raft_consensus.cc:3037] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:21:28.452522  4688 raft_consensus.cc:2454] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:28.452625  4819 raft_consensus.cc:2454] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 1.
I20240503 08:21:28.452966  4492 leader_election.cc:302] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3; no voters: 
I20240503 08:21:28.453409  5044 raft_consensus.cc:2781] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:21:28.453576  5044 raft_consensus.cc:686] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Becoming Leader. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Running, Role: LEADER
I20240503 08:21:28.453853  5044 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:28.456544  4414 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129). New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:28.756654  5173 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:28.764647  5173 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:28.809787  5044 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:28.814355  5044 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:21:28.868544  5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:28.868899  5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:28.876570  5205 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:21:28.951946  5213 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:21:28.965153  5044 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 203.852 ms
  time per row: 0.101926 ms
I20240503 08:21:29.239732  2246 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
/tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.49.132:45987
--local_ip_for_outbound_sockets=127.2.49.132
--tserver_master_addrs=127.2.49.190:33685
--webserver_port=42111
--webserver_interface=127.2.49.132
--builtin_ntp_servers=127.2.49.148:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:21:29.607856  5264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:21:29.608157  5264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:21:29.608301  5264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:21:29.616432  5264 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:21:29.616567  5264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:21:29.616745  5264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.49.132
I20240503 08:21:29.626569  5264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.49.148:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.49.132:45987
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.49.132
--webserver_port=42111
--tserver_master_addrs=127.2.49.190:33685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.49.132
--log_dir=/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:21:29.628088  5264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:21:29.629887  5264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:21:29.643595  5269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:29.643647  5272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:21:29.643748  5270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:21:29.648136  5264 server_base.cc:1034] running on GCE node
I20240503 08:21:29.648792  5264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:21:29.649784  5264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:21:29.650982  5264 hybrid_clock.cc:648] HybridClock initialized: now 1714724489650921 us; error 56 us; skew 500 ppm
I20240503 08:21:29.651364  5264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:21:29.653702  5264 webserver.cc:460] Webserver started at http://127.2.49.132:42111/ using document root <none> and password file <none>
I20240503 08:21:29.654573  5264 fs_manager.cc:359] Metadata directory not provided
I20240503 08:21:29.654701  5264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:21:29.660051  5264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240503 08:21:29.663168  5278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:29.664635  5264 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 08:21:29.664814  5264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ac08ff0f1c2340389ce4527584c90c29"
format_stamp: "Formatted at 2024-05-03 08:21:21 on dist-test-slave-dz87"
I20240503 08:21:29.665860  5264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:21:29.695466  5264 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 08:21:29.696347  5264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:21:29.696645  5264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:21:29.697716  5264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:21:29.699427  5264 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:21:29.699532  5264 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:29.699673  5264 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:21:29.699728  5264 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 08:21:29.733913  5264 rpc_server.cc:307] RPC server started. Bound to: 127.2.49.132:45987
I20240503 08:21:29.733955  5390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.49.132:45987 every 8 connection(s)
I20240503 08:21:29.735421  5264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:21:29.744107  2246 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu as pid 5264
I20240503 08:21:29.748946  5391 heartbeater.cc:347] Connected to a master server at 127.2.49.190:33685
I20240503 08:21:29.749338  5391 heartbeater.cc:464] Registering TS with master...
I20240503 08:21:29.750468  5391 heartbeater.cc:511] Master 127.2.49.190:33685 requested a full tablet report, sending...
I20240503 08:21:29.752519  4410 ts_manager.cc:194] Re-registered known tserver with Master: ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987)
I20240503 08:21:29.753432  4410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.49.132:49391
I20240503 08:21:30.269790  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.274246  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.297621  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:30.299990  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:30.465535  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.467097  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.475582  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:30.486361  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:30.652268  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.655206  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.662834  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:30.666323  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:30.756719  5391 heartbeater.cc:503] Master 127.2.49.190:33685 was elected leader, sending a full tablet report...
I20240503 08:21:30.840468  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.840833  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:30.842711  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:30.843014  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:30.916543  4688 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:30.919322  4556 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:30.920012  5209 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:30.920238  4818 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:30.920799  5209 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:30.923736  5459 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.924398  4556 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
W20240503 08:21:30.929200  4622 consensus_peers.cc:485] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f1be98d63454a3396be3d44cc4e9578. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:30.929208  4818 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.932667  4407 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.952124  4556 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:30.955276  4688 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:30.955812  4818 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:30.956135  5546 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:30.956494  5259 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:30.960417  5546 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
W20240503 08:21:30.963513  4490 consensus_peers.cc:485] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e823e4ca6d5e49a68456ed1da6a82b59. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:30.968236  4818 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.975472  4408 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.976145  4688 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.988901  4818 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:30.991812  4688 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:30.992101  4556 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:30.992535  5058 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:30.992879  5558 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:30.996678  5058 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.999675  4556 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:30.999580  4688 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
W20240503 08:21:31.000072  4753 consensus_peers.cc:485] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9db45909d7d4de182218001125a8e6a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:31.004144  4408 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.020872  4688 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
W20240503 08:21:31.023795  4622 consensus_peers.cc:485] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 903953aea4a84187a6556e4fef71d7f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:31.023789  4818 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:31.023789  4556 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:31.024441  5532 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:31.025337  5532 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:31.027942  5209 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.028589  4818 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.029039  4556 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.033928  4407 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:31.047322  4688 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.049631  4556 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:31.049748  4818 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:21:31.049816  4622 consensus_peers.cc:485] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6950bc46aa234575ad15e3b34a7e4e3d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:31.050179  5532 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:31.050743  5532 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:31.053274  5532 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.053970  4556 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.054461  4818 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:31.059549  4406 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:31.332123  5615 ts_tablet_manager.cc:902] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:31.335069  5615 tablet_copy_client.cc:308] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:42693
I20240503 08:21:31.343334  4576 tablet_copy_service.cc:140] P 95e0be617bff4a0b9a672124c6ac4f1e: Received BeginTabletCopySession request for tablet e823e4ca6d5e49a68456ed1da6a82b59 from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:58223)
I20240503 08:21:31.343694  4576 tablet_copy_service.cc:161] P 95e0be617bff4a0b9a672124c6ac4f1e: Beginning new tablet copy session on tablet e823e4ca6d5e49a68456ed1da6a82b59 from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:58223: session id = ac08ff0f1c2340389ce4527584c90c29-e823e4ca6d5e49a68456ed1da6a82b59
I20240503 08:21:31.347373  4576 tablet_copy_source_session.cc:205] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:31.353120  5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e823e4ca6d5e49a68456ed1da6a82b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:31.365612  5615 tablet_copy_client.cc:783] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:31.365964  5615 tablet_copy_client.cc:647] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:31.368570  5615 tablet_copy_client.cc:523] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:31.372165  5615 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:31.381764  5615 log.cc:824] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Log is configured to *not* fsync() on all Append() calls
I20240503 08:21:31.414453  5615 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:31.414984  5615 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:31.415419  5615 ts_tablet_manager.cc:1367] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.043s	user 0.028s	sys 0.016s
I20240503 08:21:31.420315  5615 raft_consensus.cc:348] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.420769  5615 raft_consensus.cc:729] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:31.421278  5615 consensus_queue.cc:260] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.422870  5615 ts_tablet_manager.cc:1398] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20240503 08:21:31.423930  4576 tablet_copy_service.cc:342] P 95e0be617bff4a0b9a672124c6ac4f1e: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-e823e4ca6d5e49a68456ed1da6a82b59 received from {username='slave'} at 127.2.49.132:58223
I20240503 08:21:31.424140  4576 tablet_copy_service.cc:433] P 95e0be617bff4a0b9a672124c6ac4f1e: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-e823e4ca6d5e49a68456ed1da6a82b59 on tablet e823e4ca6d5e49a68456ed1da6a82b59 with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:31.435365  5615 ts_tablet_manager.cc:902] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:31.436431  5615 tablet_copy_client.cc:308] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:31.441905  4708 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 6950bc46aa234575ad15e3b34a7e4e3d from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:39977)
I20240503 08:21:31.442191  4708 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 6950bc46aa234575ad15e3b34a7e4e3d from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:39977: session id = ac08ff0f1c2340389ce4527584c90c29-6950bc46aa234575ad15e3b34a7e4e3d
I20240503 08:21:31.445142  4708 tablet_copy_source_session.cc:205] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:31.447037  5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6950bc46aa234575ad15e3b34a7e4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:31.451524  5615 tablet_copy_client.cc:783] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:31.451833  5615 tablet_copy_client.cc:647] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:31.453917  5615 tablet_copy_client.cc:523] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:31.456401  5615 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:31.485357  5643 ts_tablet_manager.cc:902] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:31.487788  5643 tablet_copy_client.cc:308] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:31.487840  5615 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:31.488297  5615 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:31.488592  5615 ts_tablet_manager.cc:1367] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.032s	user 0.027s	sys 0.005s
I20240503 08:21:31.488638  4708 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 8f1be98d63454a3396be3d44cc4e9578 from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:39977)
I20240503 08:21:31.488858  4708 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 8f1be98d63454a3396be3d44cc4e9578 from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:39977: session id = ac08ff0f1c2340389ce4527584c90c29-8f1be98d63454a3396be3d44cc4e9578
I20240503 08:21:31.489441  5615 raft_consensus.cc:348] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.489646  5615 raft_consensus.cc:729] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:31.489828  5615 consensus_queue.cc:260] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.490360  5615 ts_tablet_manager.cc:1398] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:21:31.491478  4708 tablet_copy_source_session.cc:205] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:31.492710  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-6950bc46aa234575ad15e3b34a7e4e3d received from {username='slave'} at 127.2.49.132:39977
I20240503 08:21:31.492862  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-6950bc46aa234575ad15e3b34a7e4e3d on tablet 6950bc46aa234575ad15e3b34a7e4e3d with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:31.492976  5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1be98d63454a3396be3d44cc4e9578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:31.494779  5615 ts_tablet_manager.cc:902] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:31.496210  5615 tablet_copy_client.cc:308] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:44865
I20240503 08:21:31.499471  5643 tablet_copy_client.cc:783] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:31.499918  5643 tablet_copy_client.cc:647] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:31.501763  5643 tablet_copy_client.cc:523] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:31.503100  4839 tablet_copy_service.cc:140] P e6f88466bd394def8c59e6d892ba6129: Received BeginTabletCopySession request for tablet a9db45909d7d4de182218001125a8e6a from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:37243)
I20240503 08:21:31.503317  4839 tablet_copy_service.cc:161] P e6f88466bd394def8c59e6d892ba6129: Beginning new tablet copy session on tablet a9db45909d7d4de182218001125a8e6a from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:37243: session id = ac08ff0f1c2340389ce4527584c90c29-a9db45909d7d4de182218001125a8e6a
I20240503 08:21:31.504400  5643 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:31.506537  4839 tablet_copy_source_session.cc:205] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:31.508152  5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9db45909d7d4de182218001125a8e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:31.512984  5615 tablet_copy_client.cc:783] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:31.513276  5615 tablet_copy_client.cc:647] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:31.515648  5615 tablet_copy_client.cc:523] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:31.518426  5615 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:31.537941  5643 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:31.538419  5643 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:31.538695  5643 ts_tablet_manager.cc:1367] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.034s	user 0.031s	sys 0.004s
I20240503 08:21:31.539495  5643 raft_consensus.cc:348] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.539727  5643 raft_consensus.cc:729] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:31.539950  5643 consensus_queue.cc:260] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.540437  5643 ts_tablet_manager.cc:1398] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:31.541471  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-8f1be98d63454a3396be3d44cc4e9578 received from {username='slave'} at 127.2.49.132:39977
I20240503 08:21:31.541615  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-8f1be98d63454a3396be3d44cc4e9578 on tablet 8f1be98d63454a3396be3d44cc4e9578 with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:31.552263  5615 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:31.552695  5615 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:31.552940  5615 ts_tablet_manager.cc:1367] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.035s	user 0.023s	sys 0.008s
I20240503 08:21:31.553572  5615 raft_consensus.cc:348] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.553761  5615 raft_consensus.cc:729] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:31.553970  5615 consensus_queue.cc:260] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.554517  5615 ts_tablet_manager.cc:1398] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240503 08:21:31.555635  4839 tablet_copy_service.cc:342] P e6f88466bd394def8c59e6d892ba6129: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-a9db45909d7d4de182218001125a8e6a received from {username='slave'} at 127.2.49.132:37243
I20240503 08:21:31.555845  4839 tablet_copy_service.cc:433] P e6f88466bd394def8c59e6d892ba6129: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-a9db45909d7d4de182218001125a8e6a on tablet a9db45909d7d4de182218001125a8e6a with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:31.600900  5615 ts_tablet_manager.cc:902] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:31.610899  5615 tablet_copy_client.cc:308] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:31.611713  4707 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 903953aea4a84187a6556e4fef71d7f8 from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:39977)
I20240503 08:21:31.611876  4707 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 903953aea4a84187a6556e4fef71d7f8 from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:39977: session id = ac08ff0f1c2340389ce4527584c90c29-903953aea4a84187a6556e4fef71d7f8
I20240503 08:21:31.614661  4707 tablet_copy_source_session.cc:205] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:31.615957  5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903953aea4a84187a6556e4fef71d7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:31.621089  5615 tablet_copy_client.cc:783] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:31.621357  5615 tablet_copy_client.cc:647] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:31.622984  5615 tablet_copy_client.cc:523] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:31.625430  5615 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:31.656508  5615 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:31.656929  5615 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:31.657186  5615 ts_tablet_manager.cc:1367] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.032s	user 0.027s	sys 0.004s
I20240503 08:21:31.657896  5615 raft_consensus.cc:348] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.658113  5615 raft_consensus.cc:729] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:31.658339  5615 consensus_queue.cc:260] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:31.658818  5615 ts_tablet_manager.cc:1398] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:21:31.659655  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-903953aea4a84187a6556e4fef71d7f8 received from {username='slave'} at 127.2.49.132:39977
I20240503 08:21:31.659806  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-903953aea4a84187a6556e4fef71d7f8 on tablet 903953aea4a84187a6556e4fef71d7f8 with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:31.824916  5346 raft_consensus.cc:1203] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:31.828905  5346 raft_consensus.cc:1203] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:31.875228  5346 raft_consensus.cc:1203] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:31.998052  5346 raft_consensus.cc:1203] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:32.102507  5346 raft_consensus.cc:1203] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:32.289300  5689 raft_consensus.cc:1050] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:32.290138  5689 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.292577  4556 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.292631  4818 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.292742  5346 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.293241  5689 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.293591  5743 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.293845  5532 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.298454  5689 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.299149  4556 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.301954  5346 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.302489  4818 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.303296  4407 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:32.309039  5743 raft_consensus.cc:1050] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:32.310015  5743 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.314425  5346 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.314399  4688 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.314787  4556 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.314960  5743 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.316072  5689 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.316244  4818 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.316396  4819 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.316773  5345 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.317313  5689 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.317447  5532 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.317767  5691 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.320827  5691 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.322175  5345 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.326304  4818 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.332369  4401 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6950bc46aa234575ad15e3b34a7e4e3d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:32.332741  4406 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.335938  5689 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.338570  4536 tablet_service.cc:1514] Processing DeleteTablet for tablet 6950bc46aa234575ad15e3b34a7e4e3d with delete_type TABLET_DATA_TOMBSTONED (TS 95e0be617bff4a0b9a672124c6ac4f1e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58448
I20240503 08:21:32.340915  4556 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.344271  4410 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:32.346765  5346 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.347426  4819 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.351744  4688 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.353750  5346 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.353979  4819 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.354555  5743 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.355067  5743 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.357270  5691 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.358314  5345 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.358537  4819 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.362893  4401 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f1be98d63454a3396be3d44cc4e9578 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:32.364210  4410 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.365190  5760 tablet_replica.cc:330] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: stopping tablet replica
I20240503 08:21:32.365702  5760 raft_consensus.cc:2229] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:32.366184  5760 raft_consensus.cc:2258] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:32.367412  5690 raft_consensus.cc:1050] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:32.368613  5760 ts_tablet_manager.cc:1875] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:32.368505  5690 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.368870  4536 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1be98d63454a3396be3d44cc4e9578 with delete_type TABLET_DATA_TOMBSTONED (TS 95e0be617bff4a0b9a672124c6ac4f1e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58448
I20240503 08:21:32.371902  5346 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.373325  4819 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.372756  5761 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:21:32.374711  5761 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240503 08:21:32.376255  5760 ts_tablet_manager.cc:1888] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:32.377871  5760 log.cc:1196] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/6950bc46aa234575ad15e3b34a7e4e3d
I20240503 08:21:32.378981  4400 catalog_manager.cc:4830] TS 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): tablet 6950bc46aa234575ad15e3b34a7e4e3d (table rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]) successfully deleted
I20240503 08:21:32.382510  4688 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.383077  5760 tablet_replica.cc:330] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: stopping tablet replica
I20240503 08:21:32.383436  5761 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.384550  5760 raft_consensus.cc:2229] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:32.384841  5690 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.385691  4819 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.390067  5760 raft_consensus.cc:2258] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:32.390499  5346 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.391206  5760 ts_tablet_manager.cc:1875] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:32.391347  4406 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:32.397468  5760 ts_tablet_manager.cc:1888] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:32.397683  5760 log.cc:1196] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f1be98d63454a3396be3d44cc4e9578
I20240503 08:21:32.398834  4400 catalog_manager.cc:4830] TS 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): tablet 8f1be98d63454a3396be3d44cc4e9578 (table rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]) successfully deleted
I20240503 08:21:32.403651  4687 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.406332  4556 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.408452  5346 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.408670  4687 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.409114  5763 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.409438  5761 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.411958  5690 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.412747  5346 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.412953  4687 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.415694  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e823e4ca6d5e49a68456ed1da6a82b59 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:32.418378  4408 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.422529  4798 tablet_service.cc:1514] Processing DeleteTablet for tablet e823e4ca6d5e49a68456ed1da6a82b59 with delete_type TABLET_DATA_TOMBSTONED (TS e6f88466bd394def8c59e6d892ba6129 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60418
I20240503 08:21:32.423631  5777 tablet_replica.cc:330] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: stopping tablet replica
I20240503 08:21:32.424171  5777 raft_consensus.cc:2229] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:32.424583  5777 raft_consensus.cc:2258] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:32.426262  5777 ts_tablet_manager.cc:1875] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:32.432425  5777 ts_tablet_manager.cc:1888] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:32.432603  5777 log.cc:1196] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/e823e4ca6d5e49a68456ed1da6a82b59
I20240503 08:21:32.433490  4398 catalog_manager.cc:4830] TS e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): tablet e823e4ca6d5e49a68456ed1da6a82b59 (table rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]) successfully deleted
I20240503 08:21:32.461616  4556 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:32.463881  4687 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:32.464185  4819 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:32.464514  5690 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:21:32.464799  4490 consensus_peers.cc:485] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a85c62c6b934706991a68b034430c3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:32.465029  5761 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:32.467969  5763 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:32.468700  4819 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:32.468645  4687 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:32.473568  4406 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index -1 to 4, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:32.544000  5811 raft_consensus.cc:1050] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:32.544888  5811 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.547561  5346 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.547561  4556 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.548218  5813 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.548485  4687 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.548599  5716 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.549113  5811 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.552789  5813 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.553496  5346 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.553637  4687 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.558467  4410 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:32.559161  5691 raft_consensus.cc:1050] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:32.559901  5691 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.559986  4556 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.562359  4819 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.562934  5689 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.565631  4556 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.565964  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:32.565963  4818 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.566141  5691 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.566764  5689 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:32.568104  4687 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.568585  5811 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.571075  5345 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.571199  5813 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.571648  5689 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.572336  4556 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.576449  5811 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.576344  4818 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.577119  4687 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.578004  4407 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.580547  5345 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.584417  4398 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9db45909d7d4de182218001125a8e6a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:32.586973  4410 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 5 to 6, VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) evicted. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.586871  5346 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.590930  4536 tablet_service.cc:1514] Processing DeleteTablet for tablet a9db45909d7d4de182218001125a8e6a with delete_type TABLET_DATA_TOMBSTONED (TS 95e0be617bff4a0b9a672124c6ac4f1e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58448
I20240503 08:21:32.591328  5760 tablet_replica.cc:330] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: stopping tablet replica
I20240503 08:21:32.591543  5760 raft_consensus.cc:2229] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:32.591897  5760 raft_consensus.cc:2258] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:32.592916  5760 ts_tablet_manager.cc:1875] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:32.592763  4687 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:32.595201  4556 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.595482  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:32.595789  5532 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.596271  5689 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:32.598853  5743 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.599299  4556 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.601894  4401 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 903953aea4a84187a6556e4fef71d7f8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:32.603632  4407 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:32.605368  5760 ts_tablet_manager.cc:1888] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:32.605520  5760 log.cc:1196] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9db45909d7d4de182218001125a8e6a
I20240503 08:21:32.606284  4400 catalog_manager.cc:4830] TS 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): tablet a9db45909d7d4de182218001125a8e6a (table rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]) successfully deleted
I20240503 08:21:32.607530  4798 tablet_service.cc:1514] Processing DeleteTablet for tablet 903953aea4a84187a6556e4fef71d7f8 with delete_type TABLET_DATA_TOMBSTONED (TS e6f88466bd394def8c59e6d892ba6129 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60418
I20240503 08:21:32.607884  5777 tablet_replica.cc:330] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: stopping tablet replica
I20240503 08:21:32.608103  5777 raft_consensus.cc:2229] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:32.608369  5777 raft_consensus.cc:2258] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:32.608359  5346 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:32.609266  5777 ts_tablet_manager.cc:1875] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:32.615247  5777 ts_tablet_manager.cc:1888] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:32.615365  5777 log.cc:1196] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/903953aea4a84187a6556e4fef71d7f8
I20240503 08:21:32.615993  4398 catalog_manager.cc:4830] TS e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): tablet 903953aea4a84187a6556e4fef71d7f8 (table rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]) successfully deleted
I20240503 08:21:32.991675  5848 ts_tablet_manager.cc:902] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:32.993002  5848 tablet_copy_client.cc:308] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:42693
I20240503 08:21:32.993666  4576 tablet_copy_service.cc:140] P 95e0be617bff4a0b9a672124c6ac4f1e: Received BeginTabletCopySession request for tablet 6a85c62c6b934706991a68b034430c3b from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:58223)
I20240503 08:21:32.993829  4576 tablet_copy_service.cc:161] P 95e0be617bff4a0b9a672124c6ac4f1e: Beginning new tablet copy session on tablet 6a85c62c6b934706991a68b034430c3b from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:58223: session id = ac08ff0f1c2340389ce4527584c90c29-6a85c62c6b934706991a68b034430c3b
I20240503 08:21:32.995998  4576 tablet_copy_source_session.cc:205] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:32.997222  5848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a85c62c6b934706991a68b034430c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:33.001974  5848 tablet_copy_client.cc:783] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:33.002341  5848 tablet_copy_client.cc:647] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:33.004022  5848 tablet_copy_client.cc:523] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:33.006438  5848 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:33.035718  5848 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:33.036150  5848 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:33.036404  5848 ts_tablet_manager.cc:1367] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.030s	user 0.025s	sys 0.003s
I20240503 08:21:33.037108  5848 raft_consensus.cc:348] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:33.037328  5848 raft_consensus.cc:729] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:33.037524  5848 consensus_queue.cc:260] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:33.037992  5848 ts_tablet_manager.cc:1398] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:21:33.038884  4576 tablet_copy_service.cc:342] P 95e0be617bff4a0b9a672124c6ac4f1e: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-6a85c62c6b934706991a68b034430c3b received from {username='slave'} at 127.2.49.132:58223
I20240503 08:21:33.039041  4576 tablet_copy_service.cc:433] P 95e0be617bff4a0b9a672124c6ac4f1e: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-6a85c62c6b934706991a68b034430c3b on tablet 6a85c62c6b934706991a68b034430c3b with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:33.529861  5346 raft_consensus.cc:1203] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 08:21:34.131983  5869 raft_consensus.cc:1050] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:34.132722  5869 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:34.135336  5346 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:34.135336  4819 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:34.135728  4687 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:34.135972  5869 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:34.136325  5886 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:34.136699  5868 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:34.141147  4819 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.141219  5346 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.140481  5869 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.147456  4408 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.152602  4687 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.160674  4556 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:34.162628  5346 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:34.162756  4687 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:34.163345  5886 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:34.163697  5868 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:34.166494  5869 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.167344  5346 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.167505  4688 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:34.169668  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a85c62c6b934706991a68b034430c3b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:34.172333  4410 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 5 to 6, VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) evicted. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:34.176208  4798 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a85c62c6b934706991a68b034430c3b with delete_type TABLET_DATA_TOMBSTONED (TS e6f88466bd394def8c59e6d892ba6129 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60418
I20240503 08:21:34.176980  5895 tablet_replica.cc:330] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: stopping tablet replica
I20240503 08:21:34.177242  5895 raft_consensus.cc:2229] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:34.177556  5895 raft_consensus.cc:2258] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:34.178401  5895 ts_tablet_manager.cc:1875] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:34.183008  5895 ts_tablet_manager.cc:1888] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:21:34.183161  5895 log.cc:1196] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-2/wal/wals/6a85c62c6b934706991a68b034430c3b
I20240503 08:21:34.183966  4398 catalog_manager.cc:4830] TS e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): tablet 6a85c62c6b934706991a68b034430c3b (table rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]) successfully deleted
I20240503 08:21:34.313678  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:34.326886  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:34.337257  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:34.342729  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:34.492123  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:34.494660  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:34.499578  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:34.501325  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:34.656340  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:34.656929  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:34.661392  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:34.668233  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:35.268110  4406 ts_manager.cc:292] Set tserver state for aaa0775f07344df4993733706f4a5bb3 to MAINTENANCE_MODE
I20240503 08:21:36.780313  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:36.780295  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:36.796703  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:36.809551  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:36.967527  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:36.976446  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:36.980410  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:36.986837  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:37.153431  4536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:37.153896  4667 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:21:37.162046  5326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:37.179844  4798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:37.249130  4819 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:37.251787  4688 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:37.251824  5346 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:37.252485  5855 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:37.252923  6120 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:37.255862  5855 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.257205  5346 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.258883  4688 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.261844  4410 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:37.272418  4688 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:37.275305  5346 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:37.276039  4819 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:37.276683  5743 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:37.277454  5743 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:37.287977  4818 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:37.291337  4688 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:37.291426  5345 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:21:37.291941  4754 consensus_peers.cc:485] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 -> Peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): Couldn't send request to peer 95e0be617bff4a0b9a672124c6ac4f1e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:37.292083  6120 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:37.292233  5743 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.292735  6123 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:37.295425  6123 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.295522  4819 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.296708  5346 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.297436  5345 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.298637  4688 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.300467  4406 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:37.303200  4410 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.306579  4398 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a9db45909d7d4de182218001125a8e6a with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:37.307848  4688 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
W20240503 08:21:37.310828  4623 consensus_peers.cc:485] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 -> Peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): Couldn't send request to peer 95e0be617bff4a0b9a672124c6ac4f1e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:37.311064  5346 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:37.311666  6138 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:37.314424  4819 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:37.315063  6138 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:37.316990  5346 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.316316  6132 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.319154  4819 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.321628  4406 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.323391  4688 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:37.323828  4401 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8f1be98d63454a3396be3d44cc4e9578 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:37.327922  4819 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:37.327922  5346 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:37.328490  5743 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:37.328820  6138 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:37.333904  6132 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.334595  5346 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.337114  4819 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:37.340209  4407 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:37.344750  4688 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:37.346951  5346 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:21:37.347246  4623 consensus_peers.cc:485] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 -> Peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): Couldn't send request to peer 95e0be617bff4a0b9a672124c6ac4f1e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:37.347919  6132 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:37.350595  6144 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.351322  5346 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.352901  4818 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:37.353914  6132 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:37.356492  4401 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6950bc46aa234575ad15e3b34a7e4e3d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:37.356065  4819 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.357041  4406 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 7 to 8, NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } } }
I20240503 08:21:37.701412  6178 ts_tablet_manager.cc:902] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Initiating tablet copy from peer e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:37.703256  6178 tablet_copy_client.cc:308] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.131:44865
I20240503 08:21:37.704119  4839 tablet_copy_service.cc:140] P e6f88466bd394def8c59e6d892ba6129: Received BeginTabletCopySession request for tablet a9db45909d7d4de182218001125a8e6a from peer 95e0be617bff4a0b9a672124c6ac4f1e ({username='slave'} at 127.2.49.129:58237)
I20240503 08:21:37.704293  4839 tablet_copy_service.cc:161] P e6f88466bd394def8c59e6d892ba6129: Beginning new tablet copy session on tablet a9db45909d7d4de182218001125a8e6a from peer 95e0be617bff4a0b9a672124c6ac4f1e at {username='slave'} at 127.2.49.129:58237: session id = 95e0be617bff4a0b9a672124c6ac4f1e-a9db45909d7d4de182218001125a8e6a
I20240503 08:21:37.706339  4839 tablet_copy_source_session.cc:205] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:37.707731  6178 ts_tablet_manager.cc:1875] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:37.712435  6178 ts_tablet_manager.cc:1888] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:37.712730  6178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9db45909d7d4de182218001125a8e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:37.715214  6178 tablet_copy_client.cc:783] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:37.715528  6178 tablet_copy_client.cc:647] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:37.717667  6178 tablet_copy_client.cc:523] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:37.720712  6178 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:37.756546  6178 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:37.757005  6178 tablet_bootstrap.cc:492] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap complete.
I20240503 08:21:37.757287  6178 ts_tablet_manager.cc:1367] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.037s	user 0.035s	sys 0.000s
I20240503 08:21:37.758014  6178 raft_consensus.cc:348] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:37.758287  6178 raft_consensus.cc:729] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: LEARNER
I20240503 08:21:37.758486  6178 consensus_queue.cc:260] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:37.759081  6178 ts_tablet_manager.cc:1398] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:21:37.759966  4839 tablet_copy_service.cc:342] P e6f88466bd394def8c59e6d892ba6129: Request end of tablet copy session 95e0be617bff4a0b9a672124c6ac4f1e-a9db45909d7d4de182218001125a8e6a received from {username='slave'} at 127.2.49.129:58237
I20240503 08:21:37.760089  4839 tablet_copy_service.cc:433] P e6f88466bd394def8c59e6d892ba6129: ending tablet copy session 95e0be617bff4a0b9a672124c6ac4f1e-a9db45909d7d4de182218001125a8e6a on tablet a9db45909d7d4de182218001125a8e6a with peer 95e0be617bff4a0b9a672124c6ac4f1e
I20240503 08:21:37.880364  6178 ts_tablet_manager.cc:902] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:37.881726  6178 tablet_copy_client.cc:308] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:37.882534  4707 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 8f1be98d63454a3396be3d44cc4e9578 from peer 95e0be617bff4a0b9a672124c6ac4f1e ({username='slave'} at 127.2.49.129:40911)
I20240503 08:21:37.882715  4707 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 8f1be98d63454a3396be3d44cc4e9578 from peer 95e0be617bff4a0b9a672124c6ac4f1e at {username='slave'} at 127.2.49.129:40911: session id = 95e0be617bff4a0b9a672124c6ac4f1e-8f1be98d63454a3396be3d44cc4e9578
I20240503 08:21:37.884697  4707 tablet_copy_source_session.cc:205] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:37.885731  6178 ts_tablet_manager.cc:1875] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:37.890455  6178 ts_tablet_manager.cc:1888] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:37.890758  6178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1be98d63454a3396be3d44cc4e9578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:37.892940  6178 tablet_copy_client.cc:783] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:37.893213  6178 tablet_copy_client.cc:647] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:37.895071  6178 tablet_copy_client.cc:523] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:37.897460  6178 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:37.930492  6178 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:37.930907  6178 tablet_bootstrap.cc:492] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap complete.
I20240503 08:21:37.931138  6178 ts_tablet_manager.cc:1367] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.034s	user 0.031s	sys 0.004s
I20240503 08:21:37.931824  6178 raft_consensus.cc:348] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:37.932034  6178 raft_consensus.cc:729] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: LEARNER
I20240503 08:21:37.932224  6178 consensus_queue.cc:260] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:37.932638  6178 ts_tablet_manager.cc:1398] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 08:21:37.933554  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session 95e0be617bff4a0b9a672124c6ac4f1e-8f1be98d63454a3396be3d44cc4e9578 received from {username='slave'} at 127.2.49.129:40911
I20240503 08:21:37.933717  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session 95e0be617bff4a0b9a672124c6ac4f1e-8f1be98d63454a3396be3d44cc4e9578 on tablet 8f1be98d63454a3396be3d44cc4e9578 with peer 95e0be617bff4a0b9a672124c6ac4f1e
I20240503 08:21:37.961215  6178 ts_tablet_manager.cc:902] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:37.962471  6178 tablet_copy_client.cc:308] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:37.963279  4707 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 6950bc46aa234575ad15e3b34a7e4e3d from peer 95e0be617bff4a0b9a672124c6ac4f1e ({username='slave'} at 127.2.49.129:40911)
I20240503 08:21:37.963435  4707 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 6950bc46aa234575ad15e3b34a7e4e3d from peer 95e0be617bff4a0b9a672124c6ac4f1e at {username='slave'} at 127.2.49.129:40911: session id = 95e0be617bff4a0b9a672124c6ac4f1e-6950bc46aa234575ad15e3b34a7e4e3d
I20240503 08:21:37.965421  4707 tablet_copy_source_session.cc:205] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:37.966506  6178 ts_tablet_manager.cc:1875] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:37.970942  6178 ts_tablet_manager.cc:1888] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:37.971227  6178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6950bc46aa234575ad15e3b34a7e4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:37.973359  6178 tablet_copy_client.cc:783] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:37.973625  6178 tablet_copy_client.cc:647] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:37.975350  6178 tablet_copy_client.cc:523] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:37.977795  6178 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap starting.
I20240503 08:21:38.009485  6178 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:38.009923  6178 tablet_bootstrap.cc:492] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Bootstrap complete.
I20240503 08:21:38.010223  6178 ts_tablet_manager.cc:1367] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent bootstrapping tablet: real 0.032s	user 0.034s	sys 0.000s
I20240503 08:21:38.010933  6178 raft_consensus.cc:348] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:38.011152  6178 raft_consensus.cc:729] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Initialized, Role: LEARNER
I20240503 08:21:38.011394  6178 consensus_queue.cc:260] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: NON_VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: true } }
I20240503 08:21:38.011962  6178 ts_tablet_manager.cc:1398] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:38.012859  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session 95e0be617bff4a0b9a672124c6ac4f1e-6950bc46aa234575ad15e3b34a7e4e3d received from {username='slave'} at 127.2.49.129:40911
I20240503 08:21:38.013015  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session 95e0be617bff4a0b9a672124c6ac4f1e-6950bc46aa234575ad15e3b34a7e4e3d on tablet 6950bc46aa234575ad15e3b34a7e4e3d with peer 95e0be617bff4a0b9a672124c6ac4f1e
I20240503 08:21:38.145364  4556 raft_consensus.cc:1203] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:21:38.324141  4556 raft_consensus.cc:1203] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:21:38.345368  4556 raft_consensus.cc:1203] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:21:38.566102  6225 raft_consensus.cc:1050] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129: attempting to promote NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e to VOTER
I20240503 08:21:38.566851  6225 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:38.569123  5346 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.569259  4556 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.569264  4688 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.569739  6225 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.570204  6258 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.570570  6226 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.575395  6258 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.576112  4688 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.576601  4556 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.580544  4407 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.581816  5346 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.589862  4819 consensus_queue.cc:237] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:38.591864  5346 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:21:38.592185  4556 raft_consensus.cc:1261] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:21:38.592376  6225 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:38.592743  6258 consensus_queue.cc:1035] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:38.594975  6226 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.595561  5346 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.597891  4398 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9db45909d7d4de182218001125a8e6a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:38.599004  4556 raft_consensus.cc:2932] T a9db45909d7d4de182218001125a8e6a P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.599964  4408 catalog_manager.cc:5484] T a9db45909d7d4de182218001125a8e6a P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:38.604025  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet a9db45909d7d4de182218001125a8e6a with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:38.604902  6267 tablet_replica.cc:330] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:38.605345  6267 raft_consensus.cc:2229] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:38.605852  6267 raft_consensus.cc:2258] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:38.607393  6267 ts_tablet_manager.cc:1875] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:38.612574  6267 ts_tablet_manager.cc:1888] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:21:38.612741  6267 log.cc:1196] T a9db45909d7d4de182218001125a8e6a P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/a9db45909d7d4de182218001125a8e6a
I20240503 08:21:38.613633  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet a9db45909d7d4de182218001125a8e6a (table rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]) successfully deleted
I20240503 08:21:38.844327  6132 raft_consensus.cc:1050] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e to VOTER
I20240503 08:21:38.845183  6132 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:38.847682  5346 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.847823  4556 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.848024  4819 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.848322  6132 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.848673  6284 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.848963  6285 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.852380  6132 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.853451  4819 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.854568  5346 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.857290  4556 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.858085  4410 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:38.924927  6144 raft_consensus.cc:1050] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER 95e0be617bff4a0b9a672124c6ac4f1e to VOTER
I20240503 08:21:38.925666  6144 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:38.928048  4556 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.928041  4819 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.928045  5346 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:38.928640  6285 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.928937  6144 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.929195  6284 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:38.932614  6285 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.933399  4556 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.936502  5346 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.937209  4819 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:38.937829  4408 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 8 to 9, 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:38.980021  4556 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:38.982259  5346 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:38.982303  4688 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:38.982899  6224 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:38.983239  6300 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:38.986469  6224 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:38.986989  4688 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:38.989926  5346 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:38.990974  4408 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:38.996028  4556 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:38.998309  4688 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:21:38.998947  4489 consensus_peers.cc:485] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e -> Peer e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): Couldn't send request to peer e6f88466bd394def8c59e6d892ba6129. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:38.999152  5346 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:38.999202  6303 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:38.999689  6224 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:39.002135  6303 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:39.002781  5346 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:39.005833  4400 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e823e4ca6d5e49a68456ed1da6a82b59 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:39.007406  4688 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:39.008006  4408 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:39.029943  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34914
I20240503 08:21:39.030232  4688 raft_consensus.cc:595] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:39.046687  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34938
I20240503 08:21:39.046883  4688 raft_consensus.cc:595] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:39.274674  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34958
I20240503 08:21:39.274883  4688 raft_consensus.cc:595] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:39.292055  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34978
I20240503 08:21:39.292256  4688 raft_consensus.cc:595] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:39.341897  6132 raft_consensus.cc:979] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: : Instructing follower ac08ff0f1c2340389ce4527584c90c29 to start an election
I20240503 08:21:39.342092  6132 raft_consensus.cc:1067] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Signalling peer ac08ff0f1c2340389ce4527584c90c29 to start an election
I20240503 08:21:39.343156  5346 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578"
dest_uuid: "ac08ff0f1c2340389ce4527584c90c29"
 from {username='slave'} at 127.2.49.130:53617
I20240503 08:21:39.343500  5346 raft_consensus.cc:482] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:21:39.343690  5346 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.345726  5346 raft_consensus.cc:504] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.347535  5346 leader_election.cc:288] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [CANDIDATE]: Term 2 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865), 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:39.347790  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "ac08ff0f1c2340389ce4527584c90c29" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:39.347880  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "ac08ff0f1c2340389ce4527584c90c29" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:39.347959  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1be98d63454a3396be3d44cc4e9578" candidate_uuid: "ac08ff0f1c2340389ce4527584c90c29" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:39.348052  4688 raft_consensus.cc:3032] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:21:39.348137  4819 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.348160  4688 raft_consensus.cc:729] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:39.348167  4556 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.348489  4688 consensus_queue.cc:260] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.349428  4688 raft_consensus.cc:3037] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.351239  4556 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac08ff0f1c2340389ce4527584c90c29 in term 2.
I20240503 08:21:39.353003  4688 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac08ff0f1c2340389ce4527584c90c29 in term 2.
I20240503 08:21:39.353539  5282 leader_election.cc:302] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, aaa0775f07344df4993733706f4a5bb3, ac08ff0f1c2340389ce4527584c90c29; no voters: 
I20240503 08:21:39.354009  6264 raft_consensus.cc:2781] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:21:39.354334  4819 raft_consensus.cc:2454] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac08ff0f1c2340389ce4527584c90c29 in term 2.
I20240503 08:21:39.354758  6264 raft_consensus.cc:686] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 2 LEADER]: Becoming Leader. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Running, Role: LEADER
I20240503 08:21:39.355343  6264 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.359450  4406 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: term changed from 1 to 2, leader changed from aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) to ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132). New cstate: current_term: 2 leader_uuid: "ac08ff0f1c2340389ce4527584c90c29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:39.360626  6285 raft_consensus.cc:979] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: : Instructing follower e6f88466bd394def8c59e6d892ba6129 to start an election
I20240503 08:21:39.360800  6285 raft_consensus.cc:1067] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Signalling peer e6f88466bd394def8c59e6d892ba6129 to start an election
I20240503 08:21:39.361588  4819 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d"
dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
 from {username='slave'} at 127.2.49.130:50669
I20240503 08:21:39.361915  4819 raft_consensus.cc:482] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:21:39.362110  4819 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.364883  4819 raft_consensus.cc:504] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.366366  5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ac08ff0f1c2340389ce4527584c90c29"
I20240503 08:21:39.366446  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:39.366650  4556 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.366673  5346 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.369534  4556 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 2.
I20240503 08:21:39.369554  5346 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 2.
I20240503 08:21:39.370146  4753 leader_election.cc:302] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, ac08ff0f1c2340389ce4527584c90c29, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:39.370272  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6950bc46aa234575ad15e3b34a7e4e3d" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:39.370478  4688 raft_consensus.cc:3032] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:21:39.370582  6258 raft_consensus.cc:2781] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:21:39.370613  4688 raft_consensus.cc:729] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:39.371031  6258 raft_consensus.cc:686] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 2 LEADER]: Becoming Leader. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Running, Role: LEADER
I20240503 08:21:39.371018  4688 consensus_queue.cc:260] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.371457  4688 raft_consensus.cc:3037] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:39.371357  6258 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.374205  4688 raft_consensus.cc:2454] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 2.
I20240503 08:21:39.374576  4819 leader_election.cc:288] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 2 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987), 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:39.374709  4406 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 reported cstate change: term changed from 1 to 2, leader changed from aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) to e6f88466bd394def8c59e6d892ba6129 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:39.463843  6356 ts_tablet_manager.cc:902] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Initiating tablet copy from peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:39.465783  6356 tablet_copy_client.cc:308] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:42693
I20240503 08:21:39.466704  4576 tablet_copy_service.cc:140] P 95e0be617bff4a0b9a672124c6ac4f1e: Received BeginTabletCopySession request for tablet e823e4ca6d5e49a68456ed1da6a82b59 from peer e6f88466bd394def8c59e6d892ba6129 ({username='slave'} at 127.2.49.131:49007)
I20240503 08:21:39.466861  4576 tablet_copy_service.cc:161] P 95e0be617bff4a0b9a672124c6ac4f1e: Beginning new tablet copy session on tablet e823e4ca6d5e49a68456ed1da6a82b59 from peer e6f88466bd394def8c59e6d892ba6129 at {username='slave'} at 127.2.49.131:49007: session id = e6f88466bd394def8c59e6d892ba6129-e823e4ca6d5e49a68456ed1da6a82b59
I20240503 08:21:39.468788  4576 tablet_copy_source_session.cc:205] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:39.470147  6356 ts_tablet_manager.cc:1875] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:39.474929  6356 ts_tablet_manager.cc:1888] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:39.475222  6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e823e4ca6d5e49a68456ed1da6a82b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:39.477597  6356 tablet_copy_client.cc:783] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:39.477898  6356 tablet_copy_client.cc:647] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:39.479923  6356 tablet_copy_client.cc:523] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:39.482831  6356 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:39.524336  6356 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:39.524766  6356 tablet_bootstrap.cc:492] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Bootstrap complete.
I20240503 08:21:39.525072  6356 ts_tablet_manager.cc:1367] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.042s	user 0.035s	sys 0.004s
I20240503 08:21:39.525925  6356 raft_consensus.cc:348] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:39.526275  6356 raft_consensus.cc:729] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: LEARNER
I20240503 08:21:39.526526  6356 consensus_queue.cc:260] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:39.527092  6356 ts_tablet_manager.cc:1398] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:21:39.527877  4576 tablet_copy_service.cc:342] P 95e0be617bff4a0b9a672124c6ac4f1e: Request end of tablet copy session e6f88466bd394def8c59e6d892ba6129-e823e4ca6d5e49a68456ed1da6a82b59 received from {username='slave'} at 127.2.49.131:49007
I20240503 08:21:39.528044  4576 tablet_copy_service.cc:433] P 95e0be617bff4a0b9a672124c6ac4f1e: ending tablet copy session e6f88466bd394def8c59e6d892ba6129-e823e4ca6d5e49a68456ed1da6a82b59 on tablet e823e4ca6d5e49a68456ed1da6a82b59 with peer e6f88466bd394def8c59e6d892ba6129
I20240503 08:21:39.752452  4556 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:39.753121  6258 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:39.838021  4688 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:39.838621  6258 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:39.842913  5346 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:39.843670  6395 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:39.847467  4819 raft_consensus.cc:1203] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:21:39.910333  4819 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Refusing update from remote peer ac08ff0f1c2340389ce4527584c90c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:39.911082  4556 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Refusing update from remote peer ac08ff0f1c2340389ce4527584c90c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:39.911069  6264 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:39.911788  6264 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:39.916170  4688 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Refusing update from remote peer ac08ff0f1c2340389ce4527584c90c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:39.918262  6401 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:39.929526  5346 consensus_queue.cc:237] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:39.931804  4819 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Refusing update from remote peer ac08ff0f1c2340389ce4527584c90c29: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:21:39.931838  4556 raft_consensus.cc:1261] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Refusing update from remote peer ac08ff0f1c2340389ce4527584c90c29: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:21:39.932432  6402 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:21:39.932723  6404 consensus_queue.cc:1035] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:21:39.935003  6403 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:39.935562  4819 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:39.935768  4556 raft_consensus.cc:2932] T 8f1be98d63454a3396be3d44cc4e9578 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:39.938781  4399 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f1be98d63454a3396be3d44cc4e9578 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:39.941011  4406 catalog_manager.cc:5484] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "ac08ff0f1c2340389ce4527584c90c29" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:39.945813  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1be98d63454a3396be3d44cc4e9578 with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:39.946564  6410 tablet_replica.cc:330] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:39.947280  6410 raft_consensus.cc:2229] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:39.947510  6410 raft_consensus.cc:2258] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:39.948336  6410 ts_tablet_manager.cc:1875] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:39.952848  6410 ts_tablet_manager.cc:1888] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:21:39.952986  6410 log.cc:1196] T 8f1be98d63454a3396be3d44cc4e9578 P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f1be98d63454a3396be3d44cc4e9578
I20240503 08:21:39.953795  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet 8f1be98d63454a3396be3d44cc4e9578 (table rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]) successfully deleted
I20240503 08:21:40.209127  4556 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:40.211601  5346 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:40.211674  4688 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:40.212265  6398 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:40.212615  6399 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:40.215464  6398 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.216946  4688 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.219179  5346 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.222990  4410 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.229197  4556 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:40.231385  4688 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:21:40.231812  4489 consensus_peers.cc:485] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e -> Peer e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): Couldn't send request to peer e6f88466bd394def8c59e6d892ba6129. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:40.231938  5346 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:40.231992  6398 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:40.232442  6399 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:40.235231  6300 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.236368  5346 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.236472  4688 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.240097  4400 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6a85c62c6b934706991a68b034430c3b with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:40.242425  4408 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.259824  6300 raft_consensus.cc:1050] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e: attempting to promote NON_VOTER e6f88466bd394def8c59e6d892ba6129 to VOTER
I20240503 08:21:40.260742  6300 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:40.263383  5346 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:40.263568  4819 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:40.264132  6399 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:40.264210  4688 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:40.264837  6399 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:40.265157  6398 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:40.268822  6398 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.269804  5346 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.272573  4688 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.273298  4819 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.275449  4406 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.281628  4556 consensus_queue.cc:237] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:40.283984  4819 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:21:40.283965  5346 raft_consensus.cc:1261] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:21:40.284570  6300 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:40.284876  6398 consensus_queue.cc:1035] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:40.287442  6399 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.288105  4819 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.291548  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e823e4ca6d5e49a68456ed1da6a82b59 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:40.292264  5346 raft_consensus.cc:2932] T e823e4ca6d5e49a68456ed1da6a82b59 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:40.293290  4410 catalog_manager.cc:5484] T e823e4ca6d5e49a68456ed1da6a82b59 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:40.297060  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet e823e4ca6d5e49a68456ed1da6a82b59 with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:40.297849  6410 tablet_replica.cc:330] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:40.298079  6410 raft_consensus.cc:2229] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:40.298370  6410 raft_consensus.cc:2258] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:40.299353  6410 ts_tablet_manager.cc:1875] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:40.303941  6410 ts_tablet_manager.cc:1888] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:21:40.304064  6410 log.cc:1196] T e823e4ca6d5e49a68456ed1da6a82b59 P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/e823e4ca6d5e49a68456ed1da6a82b59
I20240503 08:21:40.304893  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet e823e4ca6d5e49a68456ed1da6a82b59 (table rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]) successfully deleted
I20240503 08:21:40.353598  4819 consensus_queue.cc:237] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } }
I20240503 08:21:40.355896  4556 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:21:40.355896  5346 raft_consensus.cc:1261] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:21:40.356607  6454 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:21:40.356987  6453 consensus_queue.cc:1035] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:21:40.359941  6395 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:40.360743  4556 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:40.362468  5346 raft_consensus.cc:2932] T 6950bc46aa234575ad15e3b34a7e4e3d P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } } }
I20240503 08:21:40.363888  4398 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6950bc46aa234575ad15e3b34a7e4e3d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:40.365962  4406 catalog_manager.cc:5484] T 6950bc46aa234575ad15e3b34a7e4e3d P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:40.370263  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 6950bc46aa234575ad15e3b34a7e4e3d with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:40.370702  6410 tablet_replica.cc:330] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:40.370918  6410 raft_consensus.cc:2229] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:40.371150  6410 raft_consensus.cc:2258] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:40.372037  6410 ts_tablet_manager.cc:1875] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:40.376505  6410 ts_tablet_manager.cc:1888] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:21:40.376637  6410 log.cc:1196] T 6950bc46aa234575ad15e3b34a7e4e3d P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/6950bc46aa234575ad15e3b34a7e4e3d
I20240503 08:21:40.377399  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet 6950bc46aa234575ad15e3b34a7e4e3d (table rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]) successfully deleted
I20240503 08:21:40.721494  4688 consensus_queue.cc:237] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:40.723821  4819 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:40.723922  4556 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:40.724447  6477 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:40.724876  6405 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:40.728006  6405 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:40.728709  4819 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:40.734516  4556 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:40.734644  4410 catalog_manager.cc:5484] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } }
I20240503 08:21:40.740495  4688 consensus_queue.cc:237] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
W20240503 08:21:40.743786  4622 consensus_peers.cc:485] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3175f646e3db4b0fab2f78f2e7e1e973. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:40.744344  4556 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:40.745071  6477 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:40.751613  4818 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:40.752465  6481 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:40.754817  6484 ts_tablet_manager.cc:902] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Initiating tablet copy from peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:40.754519  4818 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:40.759279  6405 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:40.760421  4410 catalog_manager.cc:5484] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:40.763974  4401 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3175f646e3db4b0fab2f78f2e7e1e973 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:40.765046  4556 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:40.772941  4688 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:40.775071  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:40.775086  4556 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:40.775586  6481 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:40.775910  6405 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:40.779584  6484 tablet_copy_client.cc:308] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:42693
I20240503 08:21:40.779590  4556 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.780246  4576 tablet_copy_service.cc:140] P 95e0be617bff4a0b9a672124c6ac4f1e: Received BeginTabletCopySession request for tablet 6a85c62c6b934706991a68b034430c3b from peer e6f88466bd394def8c59e6d892ba6129 ({username='slave'} at 127.2.49.131:49007)
I20240503 08:21:40.780419  4576 tablet_copy_service.cc:161] P 95e0be617bff4a0b9a672124c6ac4f1e: Beginning new tablet copy session on tablet 6a85c62c6b934706991a68b034430c3b from peer e6f88466bd394def8c59e6d892ba6129 at {username='slave'} at 127.2.49.131:49007: session id = e6f88466bd394def8c59e6d892ba6129-6a85c62c6b934706991a68b034430c3b
I20240503 08:21:40.780251  5346 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.778893  6405 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.783583  4576 tablet_copy_source_session.cc:205] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:40.785791  6484 ts_tablet_manager.cc:1875] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:40.785789  4408 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:40.792349  6484 ts_tablet_manager.cc:1888] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:40.792789  6484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a85c62c6b934706991a68b034430c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:40.794559  4688 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:40.795670  6484 tablet_copy_client.cc:783] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:40.795907  6484 tablet_copy_client.cc:647] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:40.797739  4556 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:21:40.797791  4621 consensus_peers.cc:485] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 -> Peer e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): Couldn't send request to peer e6f88466bd394def8c59e6d892ba6129. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:40.798204  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:21:40.798312  6481 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:40.798488  6484 tablet_copy_client.cc:523] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:40.798830  6483 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:40.800932  6477 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.801509  4556 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.802997  6484 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:40.802626  5345 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } } }
I20240503 08:21:40.807016  4401 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 903953aea4a84187a6556e4fef71d7f8 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:40.807659  4410 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 7 to 8, NON_VOTER e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) added. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:40.844935  6484 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:40.845353  6484 tablet_bootstrap.cc:492] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Bootstrap complete.
I20240503 08:21:40.845585  6484 ts_tablet_manager.cc:1367] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.043s	user 0.035s	sys 0.003s
I20240503 08:21:40.846280  6484 raft_consensus.cc:348] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:40.846503  6484 raft_consensus.cc:729] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: LEARNER
I20240503 08:21:40.846659  6484 consensus_queue.cc:260] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:40.847076  6484 ts_tablet_manager.cc:1398] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240503 08:21:40.847939  4576 tablet_copy_service.cc:342] P 95e0be617bff4a0b9a672124c6ac4f1e: Request end of tablet copy session e6f88466bd394def8c59e6d892ba6129-6a85c62c6b934706991a68b034430c3b received from {username='slave'} at 127.2.49.131:49007
I20240503 08:21:40.848078  4576 tablet_copy_service.cc:433] P 95e0be617bff4a0b9a672124c6ac4f1e: ending tablet copy session e6f88466bd394def8c59e6d892ba6129-6a85c62c6b934706991a68b034430c3b on tablet 6a85c62c6b934706991a68b034430c3b with peer e6f88466bd394def8c59e6d892ba6129
I20240503 08:21:41.127605  4818 raft_consensus.cc:1203] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:21:41.227906  6484 ts_tablet_manager.cc:902] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:41.229279  6484 tablet_copy_client.cc:308] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:41.230093  4707 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 903953aea4a84187a6556e4fef71d7f8 from peer e6f88466bd394def8c59e6d892ba6129 ({username='slave'} at 127.2.49.131:35549)
I20240503 08:21:41.230370  4707 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 903953aea4a84187a6556e4fef71d7f8 from peer e6f88466bd394def8c59e6d892ba6129 at {username='slave'} at 127.2.49.131:35549: session id = e6f88466bd394def8c59e6d892ba6129-903953aea4a84187a6556e4fef71d7f8
I20240503 08:21:41.232481  4707 tablet_copy_source_session.cc:205] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:41.233552  6484 ts_tablet_manager.cc:1875] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:21:41.238505  6484 ts_tablet_manager.cc:1888] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:21:41.238777  6484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903953aea4a84187a6556e4fef71d7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:41.241029  6484 tablet_copy_client.cc:783] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:41.241252  6484 tablet_copy_client.cc:647] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:41.243049  6484 tablet_copy_client.cc:523] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:41.243943  6533 ts_tablet_manager.cc:902] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903)
I20240503 08:21:41.245325  6533 tablet_copy_client.cc:308] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.130:44903
I20240503 08:21:41.245820  6484 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Bootstrap starting.
I20240503 08:21:41.246080  4707 tablet_copy_service.cc:140] P aaa0775f07344df4993733706f4a5bb3: Received BeginTabletCopySession request for tablet 3175f646e3db4b0fab2f78f2e7e1e973 from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:39977)
I20240503 08:21:41.246277  4707 tablet_copy_service.cc:161] P aaa0775f07344df4993733706f4a5bb3: Beginning new tablet copy session on tablet 3175f646e3db4b0fab2f78f2e7e1e973 from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:39977: session id = ac08ff0f1c2340389ce4527584c90c29-3175f646e3db4b0fab2f78f2e7e1e973
I20240503 08:21:41.249303  4707 tablet_copy_source_session.cc:205] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:41.250715  6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3175f646e3db4b0fab2f78f2e7e1e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:41.255470  6533 tablet_copy_client.cc:783] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:41.255707  6533 tablet_copy_client.cc:647] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:41.257553  6533 tablet_copy_client.cc:523] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:41.259990  6533 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:41.288266  6484 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:41.288753  6484 tablet_bootstrap.cc:492] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Bootstrap complete.
I20240503 08:21:41.289019  6484 ts_tablet_manager.cc:1367] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Time spent bootstrapping tablet: real 0.043s	user 0.042s	sys 0.000s
I20240503 08:21:41.289664  6484 raft_consensus.cc:348] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:41.289882  6484 raft_consensus.cc:729] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Initialized, Role: LEARNER
I20240503 08:21:41.290087  6484 consensus_queue.cc:260] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: NON_VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: true } }
I20240503 08:21:41.290769  6484 ts_tablet_manager.cc:1398] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:21:41.291504  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session e6f88466bd394def8c59e6d892ba6129-903953aea4a84187a6556e4fef71d7f8 received from {username='slave'} at 127.2.49.131:35549
I20240503 08:21:41.291641  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session e6f88466bd394def8c59e6d892ba6129-903953aea4a84187a6556e4fef71d7f8 on tablet 903953aea4a84187a6556e4fef71d7f8 with peer e6f88466bd394def8c59e6d892ba6129
I20240503 08:21:41.302459  6533 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:41.302943  6533 tablet_bootstrap.cc:492] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:41.303246  6533 ts_tablet_manager.cc:1367] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.043s	user 0.043s	sys 0.000s
I20240503 08:21:41.304020  6533 raft_consensus.cc:348] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:41.304311  6533 raft_consensus.cc:729] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:41.304545  6533 consensus_queue.cc:260] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:41.305872  6533 ts_tablet_manager.cc:1398] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 08:21:41.306613  4707 tablet_copy_service.cc:342] P aaa0775f07344df4993733706f4a5bb3: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-3175f646e3db4b0fab2f78f2e7e1e973 received from {username='slave'} at 127.2.49.132:39977
I20240503 08:21:41.306768  4707 tablet_copy_service.cc:433] P aaa0775f07344df4993733706f4a5bb3: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-3175f646e3db4b0fab2f78f2e7e1e973 on tablet 3175f646e3db4b0fab2f78f2e7e1e973 with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:41.682806  6528 raft_consensus.cc:1050] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e: attempting to promote NON_VOTER e6f88466bd394def8c59e6d892ba6129 to VOTER
I20240503 08:21:41.683521  6528 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:41.685992  4818 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:41.686420  5346 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:41.686911  4688 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:41.686955  6573 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:41.687649  6572 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:41.688021  6528 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:41.691833  6528 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.692516  4688 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.692518  5346 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.696952  4819 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.698205  4406 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.708527  4556 consensus_queue.cc:237] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:41.710583  4819 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:21:41.710764  5346 raft_consensus.cc:1261] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:21:41.711339  6528 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:41.711789  6570 consensus_queue.cc:1035] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:41.714563  6573 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.715341  5346 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.715647  4819 raft_consensus.cc:2932] T 6a85c62c6b934706991a68b034430c3b P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.719266  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a85c62c6b934706991a68b034430c3b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:41.720778  4407 catalog_manager.cc:5484] T 6a85c62c6b934706991a68b034430c3b P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 9 to 10, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:41.724895  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a85c62c6b934706991a68b034430c3b with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:41.725628  6580 tablet_replica.cc:330] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:41.725845  6580 raft_consensus.cc:2229] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:41.726315  6580 raft_consensus.cc:2258] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:41.727264  6580 ts_tablet_manager.cc:1875] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:41.732293  6580 ts_tablet_manager.cc:1888] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:21:41.732410  6580 log.cc:1196] T 6a85c62c6b934706991a68b034430c3b P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a85c62c6b934706991a68b034430c3b
I20240503 08:21:41.733188  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet 6a85c62c6b934706991a68b034430c3b (table rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]) successfully deleted
I20240503 08:21:41.797595  5346 raft_consensus.cc:1203] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:21:41.827008  4819 raft_consensus.cc:1203] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 08:21:41.992645  4556 consensus_queue.cc:237] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:41.994912  4819 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:41.994947  4688 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:41.995446  6570 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:41.995757  6573 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:41.998281  6572 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:41.998945  4688 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:42.001785  4819 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:42.002924  4407 catalog_manager.cc:5484] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } }
I20240503 08:21:42.008756  4556 consensus_queue.cc:237] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:42.011358  4688 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:21:42.011552  4490 consensus_peers.cc:485] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0044f42850c84c95a6543052a258ee36. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:42.011590  4819 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:42.011951  6573 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:42.012387  6572 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:42.016019  6573 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:42.016677  4688 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:42.017606  4819 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:42.020001  4400 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0044f42850c84c95a6543052a258ee36 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:42.022361  4408 catalog_manager.cc:5484] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 reported cstate change: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:42.236096  6597 raft_consensus.cc:1050] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:42.236840  6597 consensus_queue.cc:237] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:42.239638  4556 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:42.239629  5346 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:42.239823  4819 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:42.240234  6622 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:42.240643  6597 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:42.240902  6596 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:42.245265  6597 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:42.245784  4556 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:42.246680  5346 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:42.251910  4410 catalog_manager.cc:5484] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:42.252175  4819 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:42.292711  6597 raft_consensus.cc:1050] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: attempting to promote NON_VOTER e6f88466bd394def8c59e6d892ba6129 to VOTER
I20240503 08:21:42.293418  6597 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:42.295738  4819 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 LEARNER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:42.295989  4556 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:42.295995  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer aaa0775f07344df4993733706f4a5bb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:21:42.296564  6596 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:42.296965  6597 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:42.297276  6481 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:21:42.297515  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35120
I20240503 08:21:42.297698  4688 raft_consensus.cc:595] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:42.301466  6481 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:42.302214  4819 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:42.307081  5346 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:42.308485  4410 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa0775f07344df4993733706f4a5bb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:42.309557  4556 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e6f88466bd394def8c59e6d892ba6129 (127.2.49.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:42.526531  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35142
I20240503 08:21:42.526739  4688 raft_consensus.cc:595] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:42.535534  6669 ts_tablet_manager.cc:902] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:42.537026  6669 tablet_copy_client.cc:308] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:42693
I20240503 08:21:42.537761  4576 tablet_copy_service.cc:140] P 95e0be617bff4a0b9a672124c6ac4f1e: Received BeginTabletCopySession request for tablet 0044f42850c84c95a6543052a258ee36 from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:58223)
I20240503 08:21:42.537909  4576 tablet_copy_service.cc:161] P 95e0be617bff4a0b9a672124c6ac4f1e: Beginning new tablet copy session on tablet 0044f42850c84c95a6543052a258ee36 from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:58223: session id = ac08ff0f1c2340389ce4527584c90c29-0044f42850c84c95a6543052a258ee36
I20240503 08:21:42.540114  4576 tablet_copy_source_session.cc:205] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:42.541575  6669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0044f42850c84c95a6543052a258ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:42.547727  6669 tablet_copy_client.cc:783] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:42.548030  6669 tablet_copy_client.cc:647] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:42.550066  6669 tablet_copy_client.cc:523] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:42.550276  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35168
I20240503 08:21:42.550495  4688 raft_consensus.cc:595] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:42.553436  6669 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:42.585197  6669 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:42.585636  6669 tablet_bootstrap.cc:492] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:42.585884  6669 ts_tablet_manager.cc:1367] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.033s	user 0.024s	sys 0.008s
I20240503 08:21:42.586673  6669 raft_consensus.cc:348] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:42.586881  6669 raft_consensus.cc:729] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:42.587044  6669 consensus_queue.cc:260] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:42.587567  6669 ts_tablet_manager.cc:1398] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 08:21:42.588451  4576 tablet_copy_service.cc:342] P 95e0be617bff4a0b9a672124c6ac4f1e: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-0044f42850c84c95a6543052a258ee36 received from {username='slave'} at 127.2.49.132:58223
I20240503 08:21:42.588603  4576 tablet_copy_service.cc:433] P 95e0be617bff4a0b9a672124c6ac4f1e: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-0044f42850c84c95a6543052a258ee36 on tablet 0044f42850c84c95a6543052a258ee36 with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:42.674572  6596 raft_consensus.cc:979] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: : Instructing follower e6f88466bd394def8c59e6d892ba6129 to start an election
I20240503 08:21:42.674777  6596 raft_consensus.cc:1067] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Signalling peer e6f88466bd394def8c59e6d892ba6129 to start an election
I20240503 08:21:42.675572  4819 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973"
dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
 from {username='slave'} at 127.2.49.130:50669
I20240503 08:21:42.675845  4819 raft_consensus.cc:482] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:21:42.675990  4819 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.677953  4819 raft_consensus.cc:504] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:42.679337  4819 leader_election.cc:288] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 2 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693), ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987)
I20240503 08:21:42.679390  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:42.679497  5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "ac08ff0f1c2340389ce4527584c90c29"
I20240503 08:21:42.679519  4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3175f646e3db4b0fab2f78f2e7e1e973" candidate_uuid: "e6f88466bd394def8c59e6d892ba6129" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
I20240503 08:21:42.679605  4688 raft_consensus.cc:3032] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:21:42.679723  4688 raft_consensus.cc:729] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:42.679724  4556 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.679745  5346 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.680048  4688 consensus_queue.cc:260] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:42.680371  4688 raft_consensus.cc:3037] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.682426  4688 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 2.
I20240503 08:21:42.682652  5346 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 2.
I20240503 08:21:42.682667  4556 raft_consensus.cc:2454] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f88466bd394def8c59e6d892ba6129 in term 2.
I20240503 08:21:42.683173  4753 leader_election.cc:302] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: aaa0775f07344df4993733706f4a5bb3, ac08ff0f1c2340389ce4527584c90c29, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:42.683485  6453 raft_consensus.cc:2781] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:21:42.683871  6453 raft_consensus.cc:686] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 2 LEADER]: Becoming Leader. State: Replica: e6f88466bd394def8c59e6d892ba6129, State: Running, Role: LEADER
I20240503 08:21:42.684162  6453 consensus_queue.cc:237] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:42.687994  4408 catalog_manager.cc:5484] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: term changed from 1 to 2, leader changed from aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) to e6f88466bd394def8c59e6d892ba6129 (127.2.49.131). New cstate: current_term: 2 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:42.779460  4688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8"
dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47598
I20240503 08:21:42.779678  4688 raft_consensus.cc:595] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:21:42.800276  6481 raft_consensus.cc:979] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: : Instructing follower 95e0be617bff4a0b9a672124c6ac4f1e to start an election
I20240503 08:21:42.800496  6481 raft_consensus.cc:1067] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Signalling peer 95e0be617bff4a0b9a672124c6ac4f1e to start an election
I20240503 08:21:42.801283  4556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8"
dest_uuid: "95e0be617bff4a0b9a672124c6ac4f1e"
 from {username='slave'} at 127.2.49.130:51175
I20240503 08:21:42.801622  4556 raft_consensus.cc:482] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:21:42.801797  4556 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.804059  4556 raft_consensus.cc:504] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:42.805307  4556 leader_election.cc:288] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 2 election: Requested vote from peers aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903), ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987), e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865)
I20240503 08:21:42.805642  4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "aaa0775f07344df4993733706f4a5bb3"
I20240503 08:21:42.805655  4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e6f88466bd394def8c59e6d892ba6129"
I20240503 08:21:42.805691  5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903953aea4a84187a6556e4fef71d7f8" candidate_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ac08ff0f1c2340389ce4527584c90c29"
I20240503 08:21:42.805866  4688 raft_consensus.cc:3032] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:21:42.805884  4819 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.805941  5346 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.805980  4688 raft_consensus.cc:729] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aaa0775f07344df4993733706f4a5bb3, State: Running, Role: LEADER
I20240503 08:21:42.806294  4688 consensus_queue.cc:260] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:42.806805  4688 raft_consensus.cc:3037] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:21:42.808060  5346 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 2.
I20240503 08:21:42.808789  4819 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 2.
I20240503 08:21:42.809327  4489 leader_election.cc:302] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 95e0be617bff4a0b9a672124c6ac4f1e, ac08ff0f1c2340389ce4527584c90c29, e6f88466bd394def8c59e6d892ba6129; no voters: 
I20240503 08:21:42.809635  4688 raft_consensus.cc:2454] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e0be617bff4a0b9a672124c6ac4f1e in term 2.
I20240503 08:21:42.809722  6573 raft_consensus.cc:2781] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:21:42.810272  6573 raft_consensus.cc:686] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 LEADER]: Becoming Leader. State: Replica: 95e0be617bff4a0b9a672124c6ac4f1e, State: Running, Role: LEADER
I20240503 08:21:42.810595  6573 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:42.813486  4408 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: term changed from 1 to 2, leader changed from aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) to 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129). New cstate: current_term: 2 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:42.996484  5346 raft_consensus.cc:1203] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:21:43.171741  5346 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:21:43.172564  6453 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:43.220909  4688 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:43.221560  6708 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:43.275844  4555 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:21:43.276497  6453 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:43.281177  4688 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:21:43.281865  6453 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:43.287791  4819 consensus_queue.cc:237] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:43.289763  4555 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:21:43.289774  5346 raft_consensus.cc:1261] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Refusing update from remote peer e6f88466bd394def8c59e6d892ba6129: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:21:43.290405  6738 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:43.290688  6453 consensus_queue.cc:1035] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:21:43.293185  6738 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.293879  5346 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.293879  4555 raft_consensus.cc:2932] T 3175f646e3db4b0fab2f78f2e7e1e973 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.297375  4398 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3175f646e3db4b0fab2f78f2e7e1e973 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:43.299072  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:43.298959  4410 catalog_manager.cc:5484] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 6 to 8, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "e6f88466bd394def8c59e6d892ba6129" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:43.299854  6703 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:43.303495  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 3175f646e3db4b0fab2f78f2e7e1e973 with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:43.304556  4819 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:21:43.305274  6703 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:21:43.308265  6743 tablet_replica.cc:330] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:43.308478  6743 raft_consensus.cc:2229] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:43.308674  6743 raft_consensus.cc:2258] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:43.310053  6743 ts_tablet_manager.cc:1875] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:43.314819  6743 ts_tablet_manager.cc:1888] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:21:43.314955  6743 log.cc:1196] T 3175f646e3db4b0fab2f78f2e7e1e973 P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/3175f646e3db4b0fab2f78f2e7e1e973
I20240503 08:21:43.315711  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet 3175f646e3db4b0fab2f78f2e7e1e973 (table rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]) successfully deleted
I20240503 08:21:43.493266  6703 raft_consensus.cc:1050] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:43.494004  6703 consensus_queue.cc:237] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:43.496610  5346 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:43.496644  4688 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:43.496630  4819 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:43.497375  6573 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:43.497773  6703 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:43.498102  6761 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:43.502362  6573 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.503320  4819 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.503345  5346 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.508319  4688 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.508517  4408 catalog_manager.cc:5484] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 reported cstate change: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.514627  4555 consensus_queue.cc:237] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:43.516621  5346 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:43.516732  4819 raft_consensus.cc:1261] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:43.517257  6573 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:43.517612  6761 consensus_queue.cc:1035] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:43.519948  6708 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.520581  5346 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.520740  4819 raft_consensus.cc:2932] T 0044f42850c84c95a6543052a258ee36 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.524144  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0044f42850c84c95a6543052a258ee36 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:43.525215  4406 catalog_manager.cc:5484] T 0044f42850c84c95a6543052a258ee36 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:43.529420  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 0044f42850c84c95a6543052a258ee36 with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:43.530555  6743 tablet_replica.cc:330] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:43.530794  6743 raft_consensus.cc:2229] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:43.531201  6743 raft_consensus.cc:2258] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:43.532119  6743 ts_tablet_manager.cc:1875] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:43.536669  6743 ts_tablet_manager.cc:1888] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:21:43.536794  6743 log.cc:1196] T 0044f42850c84c95a6543052a258ee36 P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/0044f42850c84c95a6543052a258ee36
I20240503 08:21:43.537629  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet 0044f42850c84c95a6543052a258ee36 (table rebalance_test_table_2 [id=a15614f2cd6e4bf8bd28acdc499bb1c3]) successfully deleted
I20240503 08:21:43.703162  4555 consensus_queue.cc:237] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } }
I20240503 08:21:43.705667  4819 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:43.705669  4688 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:21:43.706459  6708 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:43.706841  6573 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:21:43.709909  6573 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:43.711879  4819 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:43.715803  4407 catalog_manager.cc:5484] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } }
I20240503 08:21:43.719259  4688 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } }
I20240503 08:21:43.721684  4555 consensus_queue.cc:237] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:43.728519  4687 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:21:43.729805  4490 consensus_peers.cc:485] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e -> Peer ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132:45987): Couldn't send request to peer ac08ff0f1c2340389ce4527584c90c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef9404adc1c94a719047974f0f1c2a22. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:43.730785  6703 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:43.734527  4819 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:21:43.737136  6573 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:43.737820  6703 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:21:43.739513  4819 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:43.741560  4400 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ef9404adc1c94a719047974f0f1c2a22 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:21:43.743230  4687 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } } }
I20240503 08:21:43.743690  4407 catalog_manager.cc:5484] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 4 to 5, NON_VOTER ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) added. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:21:43.902802  4555 consensus_queue.cc:237] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } }
I20240503 08:21:43.904644  4819 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:21:43.904661  5346 raft_consensus.cc:1261] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:21:43.905278  6761 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:21:43.905602  6703 consensus_queue.cc:1035] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:21:43.908118  6761 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P 95e0be617bff4a0b9a672124c6ac4f1e [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:43.909024  5346 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:43.908999  4819 raft_consensus.cc:2932] T 903953aea4a84187a6556e4fef71d7f8 P e6f88466bd394def8c59e6d892ba6129 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:43.913395  4407 catalog_manager.cc:5484] T 903953aea4a84187a6556e4fef71d7f8 P ac08ff0f1c2340389ce4527584c90c29 reported cstate change: config changed from index 9 to 11, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 2 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } attrs { promote: false } } }
I20240503 08:21:43.915102  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 903953aea4a84187a6556e4fef71d7f8 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:43.917143  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 903953aea4a84187a6556e4fef71d7f8 with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:43.917765  6743 tablet_replica.cc:330] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:43.917965  6743 raft_consensus.cc:2229] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:43.918212  6743 raft_consensus.cc:2258] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:43.919288  6743 ts_tablet_manager.cc:1875] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:43.924337  6743 ts_tablet_manager.cc:1888] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:21:43.924464  6743 log.cc:1196] T 903953aea4a84187a6556e4fef71d7f8 P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/903953aea4a84187a6556e4fef71d7f8
I20240503 08:21:43.925249  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet 903953aea4a84187a6556e4fef71d7f8 (table rebalance_test_table_0 [id=629f9711f7da42f5a1c92a3924eb1ca9]) successfully deleted
I20240503 08:21:44.264515  6803 ts_tablet_manager.cc:902] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: Initiating tablet copy from peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693)
I20240503 08:21:44.265882  6803 tablet_copy_client.cc:308] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.49.129:42693
I20240503 08:21:44.266894  4576 tablet_copy_service.cc:140] P 95e0be617bff4a0b9a672124c6ac4f1e: Received BeginTabletCopySession request for tablet ef9404adc1c94a719047974f0f1c2a22 from peer ac08ff0f1c2340389ce4527584c90c29 ({username='slave'} at 127.2.49.132:58223)
I20240503 08:21:44.267095  4576 tablet_copy_service.cc:161] P 95e0be617bff4a0b9a672124c6ac4f1e: Beginning new tablet copy session on tablet ef9404adc1c94a719047974f0f1c2a22 from peer ac08ff0f1c2340389ce4527584c90c29 at {username='slave'} at 127.2.49.132:58223: session id = ac08ff0f1c2340389ce4527584c90c29-ef9404adc1c94a719047974f0f1c2a22
I20240503 08:21:44.269174  4576 tablet_copy_source_session.cc:205] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:21:44.270848  6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9404adc1c94a719047974f0f1c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:21:44.275480  6803 tablet_copy_client.cc:783] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 0 data blocks...
I20240503 08:21:44.275746  6803 tablet_copy_client.cc:647] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Starting download of 1 WAL segments...
I20240503 08:21:44.277681  6803 tablet_copy_client.cc:523] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:21:44.280292  6803 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap starting.
I20240503 08:21:44.317400  6803 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:21:44.317883  6803 tablet_bootstrap.cc:492] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: Bootstrap complete.
I20240503 08:21:44.318220  6803 ts_tablet_manager.cc:1367] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: Time spent bootstrapping tablet: real 0.038s	user 0.030s	sys 0.005s
I20240503 08:21:44.319011  6803 raft_consensus.cc:348] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:44.319249  6803 raft_consensus.cc:729] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac08ff0f1c2340389ce4527584c90c29, State: Initialized, Role: LEARNER
I20240503 08:21:44.319499  6803 consensus_queue.cc:260] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: NON_VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: true } }
I20240503 08:21:44.320173  6803 ts_tablet_manager.cc:1398] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 08:21:44.321117  4576 tablet_copy_service.cc:342] P 95e0be617bff4a0b9a672124c6ac4f1e: Request end of tablet copy session ac08ff0f1c2340389ce4527584c90c29-ef9404adc1c94a719047974f0f1c2a22 received from {username='slave'} at 127.2.49.132:58223
I20240503 08:21:44.321282  4576 tablet_copy_service.cc:433] P 95e0be617bff4a0b9a672124c6ac4f1e: ending tablet copy session ac08ff0f1c2340389ce4527584c90c29-ef9404adc1c94a719047974f0f1c2a22 on tablet ef9404adc1c94a719047974f0f1c2a22 with peer ac08ff0f1c2340389ce4527584c90c29
I20240503 08:21:44.710878  5346 raft_consensus.cc:1203] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 08:21:45.332072  6703 raft_consensus.cc:1050] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e: attempting to promote NON_VOTER ac08ff0f1c2340389ce4527584c90c29 to VOTER
I20240503 08:21:45.332701  6703 consensus_queue.cc:237] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:45.334905  5346 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 LEARNER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:45.335208  4819 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:45.335258  4687 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:21:45.335552  6761 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:45.335891  6703 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:45.336251  6835 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:21:45.339776  6703 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.340687  5346 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.340930  4687 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.343982  4819 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.344624  4406 catalog_manager.cc:5484] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 5 to 6, ac08ff0f1c2340389ce4527584c90c29 (127.2.49.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa0775f07344df4993733706f4a5bb3" member_type: VOTER last_known_addr { host: "127.2.49.130" port: 44903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:45.355990  4555 consensus_queue.cc:237] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } }
I20240503 08:21:45.357900  5346 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:45.357945  4819 raft_consensus.cc:1261] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Refusing update from remote peer 95e0be617bff4a0b9a672124c6ac4f1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:21:45.358532  6835 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:45.358848  6761 consensus_queue.cc:1035] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:21:45.361124  6761 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.361783  4819 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P e6f88466bd394def8c59e6d892ba6129 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.365693  4400 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef9404adc1c94a719047974f0f1c2a22 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:21:45.365525  5345 raft_consensus.cc:2932] T ef9404adc1c94a719047974f0f1c2a22 P ac08ff0f1c2340389ce4527584c90c29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } } }
I20240503 08:21:45.366887  4408 catalog_manager.cc:5484] T ef9404adc1c94a719047974f0f1c2a22 P 95e0be617bff4a0b9a672124c6ac4f1e reported cstate change: config changed from index 6 to 7, VOTER aaa0775f07344df4993733706f4a5bb3 (127.2.49.130) evicted. New cstate: current_term: 1 leader_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "95e0be617bff4a0b9a672124c6ac4f1e" member_type: VOTER last_known_addr { host: "127.2.49.129" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6f88466bd394def8c59e6d892ba6129" member_type: VOTER last_known_addr { host: "127.2.49.131" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac08ff0f1c2340389ce4527584c90c29" member_type: VOTER last_known_addr { host: "127.2.49.132" port: 45987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:21:45.372203  4667 tablet_service.cc:1514] Processing DeleteTablet for tablet ef9404adc1c94a719047974f0f1c2a22 with delete_type TABLET_DATA_TOMBSTONED (TS aaa0775f07344df4993733706f4a5bb3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39894
I20240503 08:21:45.373023  6844 tablet_replica.cc:330] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: stopping tablet replica
I20240503 08:21:45.373225  6844 raft_consensus.cc:2229] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:21:45.373500  6844 raft_consensus.cc:2258] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:21:45.374404  6844 ts_tablet_manager.cc:1875] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:21:45.378774  6844 ts_tablet_manager.cc:1888] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:21:45.378970  6844 log.cc:1196] T ef9404adc1c94a719047974f0f1c2a22 P aaa0775f07344df4993733706f4a5bb3: Deleting WAL directory at /tmp/dist-test-taskPaEuZ9/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723866985054-2246-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef9404adc1c94a719047974f0f1c2a22
I20240503 08:21:45.379776  4401 catalog_manager.cc:4830] TS aaa0775f07344df4993733706f4a5bb3 (127.2.49.130:44903): tablet ef9404adc1c94a719047974f0f1c2a22 (table rebalance_test_table_1 [id=d7b4d6b17a1a4645aac9e91d048f5871]) successfully deleted
I20240503 08:21:45.781394  4536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:45.798629  4667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:45.807636  4798 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:45.818994  5326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:45.971832  4536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:45.975309  4667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:45.985869  5326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:45.985867  4798 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:46.141438  4536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:46.160456  4667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:46.163163  5326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:46.177970  4798 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:46.354244  4536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:46.358481  4667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:46.359369  5326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:46.363320  4798 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:46.519021  4536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:21:46.524037  5326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:21:46.525758  4667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:21:46.535485  4798 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:21:46.746189  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4474
I20240503 08:21:46.762357  4595 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:46.933907  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4474
I20240503 08:21:46.965209  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4605
I20240503 08:21:46.971625  4728 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:21:47.093927  4754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:42693: connect: Connection refused (error 111)
W20240503 08:21:47.095453  4754 consensus_peers.cc:485] T 3175f646e3db4b0fab2f78f2e7e1e973 P e6f88466bd394def8c59e6d892ba6129 -> Peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): Couldn't send request to peer 95e0be617bff4a0b9a672124c6ac4f1e. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:42693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:47.140440  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4605
I20240503 08:21:47.170990  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4736
I20240503 08:21:47.177742  4859 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:21:47.331087  5281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.49.129:42693: connect: Connection refused (error 111)
W20240503 08:21:47.334522  5281 consensus_peers.cc:485] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 -> Peer 95e0be617bff4a0b9a672124c6ac4f1e (127.2.49.129:42693): Couldn't send request to peer 95e0be617bff4a0b9a672124c6ac4f1e. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.129:42693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:47.340596  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4736
W20240503 08:21:47.365509  5279 connection.cc:535] client connection to 127.2.49.131:44865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:21:47.366142  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 5264
W20240503 08:21:47.366736  5279 consensus_peers.cc:485] T 8f1be98d63454a3396be3d44cc4e9578 P ac08ff0f1c2340389ce4527584c90c29 -> Peer e6f88466bd394def8c59e6d892ba6129 (127.2.49.131:44865): Couldn't send request to peer e6f88466bd394def8c59e6d892ba6129. Status: Network error: Client connection negotiation failed: client connection to 127.2.49.131:44865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:21:47.372606  5385 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:47.521733  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 5264
W20240503 08:21:47.549142  4401 connection.cc:535] server connection from 127.2.49.132:49391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:21:47.549432  2246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu pid 4383
I20240503 08:21:47.550716  4445 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:21:47.716926  2246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPaEuZ9/build/asan/bin/kudu with pid 4383
2024-05-03T08:21:47Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (28676 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (51515 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (640764 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>

 1 FAILED TEST